Add index-free MiniCPM scout, local QLoRA training, and measured evaluation
Tests / core (push) Canceled after 0s
Tests / core (push) Canceled after 0s
This commit is contained in:
emil28092005
@@ -0,0 +1,265 @@
|
||||
# MiniCPM5 index-free search experiment
|
||||
|
||||
Recorded September 16, 2026, on an RTX 3050 Laptop GPU with 4 GB VRAM,
|
||||
an Intel Core i7-12650H, and approximately 30 GiB usable system RAM.
|
||||
|
||||
## Conclusion
|
||||
|
||||
The index-free search harness, MCP server, and local QLoRA pipeline work, but
|
||||
this experiment has not produced a useful replacement for a coding model's own
|
||||
search. Lower validation action loss and source-verified responses did not
|
||||
translate into reliable implementation localization. The existing indexed
|
||||
MiniLM integration remains the active MCP configuration.
|
||||
|
||||
Both adapters were trained on this laptop without a hosted teacher or solver.
|
||||
Their weights remain local; this repository publishes the implementation,
|
||||
training metadata, artifact hashes, and measured development results. No paired
|
||||
Astra experiment was run, so downstream usefulness with Astra is unproven.
|
||||
|
||||
| Variant | Target hits | Correct files | Median / p95 search time |
|
||||
| --- | ---: | ---: | ---: |
|
||||
| Original model, Ollama Q4_K_M | 0 / 30 | 2 / 30 | 4.83 / 15.41 s |
|
||||
| Original model, Transformers NF4 | 0 / 30 | 4 / 30 | 16.39 / 54.43 s |
|
||||
| QLoRA adapter v1, Transformers NF4 | 0 / 30 | 8 / 30 | 19.40 / 39.98 s |
|
||||
| QLoRA adapter v2, Transformers NF4 | 0 / 30 | 3 / 30 | 32.88 / 44.77 s |
|
||||
| Fixed separate-term keyword control | 2 / 30 | 15 / 30 | 0.23 / 0.30 s |
|
||||
|
||||
The keyword control is a deliberately simple automatic baseline, not a
|
||||
measurement of Astra using grep. The model variants are not competitive with
|
||||
even this control on target retrieval or latency in this development suite.
|
||||
|
||||
## Question and setup
|
||||
|
||||
Can a local 1B model select `grep` and bounded `read` actions, then return useful
|
||||
source ranges without building an index? This experiment implements that loop
|
||||
and a small supervised adapter-training pipeline. It does not measure coding
|
||||
task success with a larger solver or compare against Astra's own search.
|
||||
|
||||
The base model is OpenBMB MiniCPM5-1B, an Apache-2.0 Llama architecture checkpoint
|
||||
with 1,080,632,832 parameters. The protocol uses native XML function calls and
|
||||
no-think framing. The harness provides a current file listing, allows six rounds,
|
||||
at most three actions per round, 512 generated tokens per round, and up to 6,000
|
||||
returned source characters. It enforces a 90-second search budget; a backend's
|
||||
current generation or filesystem operation may slightly exceed the deadline.
|
||||
|
||||
The frozen development suite has 30 English single-function localization tasks:
|
||||
10 each from Requests, Flask, and Click. Source commits, file hashes, and target
|
||||
body ranges are pinned in [`evals/live-search-v1.json`](../../evals/live-search-v1.json).
|
||||
Targets exclude leading docstrings. A hit needs at least three target body lines
|
||||
(or the whole body when shorter), and line precision penalizes broad guesses.
|
||||
Labels are not passed to the search model.
|
||||
|
||||
## First frozen baseline: Ollama Q4_K_M
|
||||
|
||||
| Measurement | Result |
|
||||
| --- | ---: |
|
||||
| Target implementations found | **0 / 30** |
|
||||
| Correct file returned | 2 / 30 |
|
||||
| Verified completed responses | 13 / 30 |
|
||||
| Exhausted search budget | 17 / 30 |
|
||||
| Median / p95 end-to-end latency | 4.83 / 15.41 s |
|
||||
| Total invalid actions / tool errors | 37 / 22 |
|
||||
| Mean rounds / tool calls | 5.37 / 4.70 |
|
||||
| Sampled peak total GPU memory | 1,213 MiB |
|
||||
|
||||
All completed responses had valid source provenance but missed the target body.
|
||||
This illustrates why successful tool execution is not a retrieval-quality metric.
|
||||
The unmodified Q4 model is not useful enough to replace the existing indexed
|
||||
retriever on this suite. A typical failure was selecting a matching comment or
|
||||
trying to finish without reading the requested range.
|
||||
|
||||
Latency excludes model warm-up. GPU memory is device-wide usage sampled once per
|
||||
second, not an exact allocator peak. This is a small sequential laptop workload,
|
||||
not a concurrency or production-serving benchmark.
|
||||
|
||||
## Reference backend baseline
|
||||
|
||||
The corrected Transformers NF4 baseline also found **0/30 targets**, with 4/30
|
||||
correct files, 13 verified completed responses, and 17 exhausted searches.
|
||||
Median latency was **16.39 s**, p95 **54.43 s**, and sampled device memory peaked
|
||||
at **2,198 MiB**. It produced 53 invalid actions and 16 tool errors. These are
|
||||
the baseline settings used for the adapter comparison; Q4 versus NF4 alone would
|
||||
confound adapter quality with the inference backend and quantization format.
|
||||
|
||||
## Keyword control
|
||||
|
||||
A separate non-neural control uses the same executor with up to 12 literal term
|
||||
searches and three 25-line reads. Windows are ranked by the sum of
|
||||
`1 / log2(2 + returned matches)` for their distinct query terms. This bounded
|
||||
match count is a heuristic, not corpus document frequency. The algorithm sees
|
||||
query text and current files only, never target labels.
|
||||
|
||||
The control was added during development after observing the model's failures.
|
||||
An initial single-OR-query version found 0/30 targets: frequent words exhausted
|
||||
the result cap before useful matches. Searching terms separately found **2/30**.
|
||||
Both runs are retained locally. This is a weak automatic keyword baseline; it
|
||||
does not represent an experienced developer or Astra using adaptive grep.
|
||||
|
||||
An idle-laptop repeat of the separate-term control preserved all per-task scores:
|
||||
**0.229 s median / 0.296 s p95**, with 11.57 tool calls per query on average.
|
||||
No GPU or model is required. Its macro line precision was only 0.77%, so even
|
||||
the two target hits do not imply an economical set of source snippets.
|
||||
|
||||
## Local adapter training
|
||||
|
||||
The selected adapter was trained with QLoRA on the original pinned checkpoint,
|
||||
using NF4 double quantization and BF16 computation. This is supervised action
|
||||
training, not a new architecture, full pretraining, or online RL.
|
||||
|
||||
| Setting or measurement | Value |
|
||||
| --- | ---: |
|
||||
| Train / validation trajectories | 256 / 32 |
|
||||
| Train / validation action examples | 826 / 104 |
|
||||
| Epochs / optimizer steps | 1 / 104 |
|
||||
| LoRA rank / alpha / dropout | 16 / 32 / 0.05 |
|
||||
| Trainable parameters | 11,206,656 |
|
||||
| Batch size / gradient accumulation | 1 / 8 |
|
||||
| Learning rate | 0.0001 |
|
||||
| Sequence limit / overlength examples dropped | 2,048 / 0 |
|
||||
| Initial / best validation action loss | 0.33653 / 0.08246 |
|
||||
| Selected checkpoint step | 100 |
|
||||
| Training time, including validation and saves | 1,206 s (20m 6s) |
|
||||
| Peak PyTorch allocated CUDA memory | 2,779 MiB |
|
||||
| Adapter safetensors size | 44,871,152 bytes |
|
||||
|
||||
The timer starts after loading, tokenization, and optimizer setup. Allocator
|
||||
memory is different from the device-wide sampler used for search runs. The
|
||||
recorded run used the laptop's existing CUDA stack; no hosted teacher was called.
|
||||
|
||||
Data comes from the [audited CodeSearchNet subset](../../docs/DATASET.md), retaining
|
||||
source and distractor provenance. Complete candidate pools are repository-disjoint
|
||||
between training and validation; Requests, Flask, and Click are excluded by name.
|
||||
Three-file synthetic repositories preserve source filenames and vary line offsets.
|
||||
An oracle executes query-term searches, source reads, and final range selections.
|
||||
Only assistant action tokens receive loss. Such examples teach protocol and
|
||||
short search sequences, not realistic repository exploration.
|
||||
|
||||
Selection used validation action loss, not the 30-task localization scores.
|
||||
The selected weights are local at `runs/minicpm5-policy-v1/best`; weights are not
|
||||
published with the repository. The adapter's base-reference metadata was
|
||||
normalized to the official model ID and revision after training, without changing
|
||||
the numerical weights. Future training runs save those portable fields directly.
|
||||
The exact training source snapshot remains beside the local run.
|
||||
|
||||
### Adapter v1 search result and data revision
|
||||
|
||||
The first adapter still found **0/30 targets**, despite improving file hits to
|
||||
8/30 and verified completions to 25/30. Invalid actions fell from 53 to 13; tool
|
||||
errors fell from 16 to 14. Median latency was 19.40 s and p95 39.98 s with the
|
||||
unmerged PEFT adapter. Better protocol execution did not produce useful target
|
||||
body retrieval. These development timings are not isolated kernel benchmarks.
|
||||
|
||||
Inspection exposed a weakness in the first oracle recipe: it required exact
|
||||
unread function boundaries after a short grep observation. That observation
|
||||
cannot reveal the function's end. The learned policy often copied the same match
|
||||
line into both read boundaries. The revised recipe instead teaches a fixed
|
||||
observable window around a match, followed by selection from the actual read.
|
||||
Two functions per file and randomized target position reduce the shortcut of
|
||||
always selecting the only function or the end of the file. One unobservable
|
||||
training candidate was rejected; validation rejected none.
|
||||
|
||||
This revision was motivated by development traces, so subsequent localization
|
||||
results remain development results, not a fresh untouched evaluation.
|
||||
|
||||
### Adapter v2 training
|
||||
|
||||
The corrected recipe retained 256 training and 32 validation trajectories
|
||||
(826 / 104 action examples). The limit increased to 2,560 tokens; all examples
|
||||
fit without truncation or dropping. The same base, LoRA configuration, learning
|
||||
rate, seed, and one-epoch schedule were used. The run took **1,297 s (21m 37s)**,
|
||||
with **2,842 MiB** peak PyTorch allocated CUDA memory. Validation action loss
|
||||
fell from **0.48502 to 0.06406**, selecting step **104**.
|
||||
|
||||
These loss values refer to the revised examples and cannot be compared directly
|
||||
with v1's validation loss. The selected adapter is local at
|
||||
`runs/minicpm5-policy-v2/best`. See `training-v2-experiment.json`,
|
||||
`training-v2-result.json`, and `adapter-v2-manifest.json` for the exact settings
|
||||
and artifact hashes. Both adapters use unmerged PEFT inference in the reference
|
||||
backend; the reported search latency is not an optimized merged-GGUF deployment.
|
||||
|
||||
The v2 evaluation process was interrupted after 28 task results had been saved.
|
||||
The remaining two tasks were resumed after verifying the frozen code, suite,
|
||||
repository, and adapter hashes, with a new warm-up. The earlier results were
|
||||
retained. GPU samples for the first segment were not persisted, so this report
|
||||
does not claim a full-run GPU peak for v2 inference.
|
||||
|
||||
### Adapter v2 search result
|
||||
|
||||
The revised adapter again found **0/30 targets** and returned the correct file
|
||||
on 3/30 tasks. Only 7 responses completed; 23 exhausted the search budget. Median
|
||||
latency was **32.88 s**, p95 **44.77 s**, with 51 invalid actions and 22 tool errors.
|
||||
The lower validation action loss did not generalize to this search workload.
|
||||
Observed failures included poor search terms, matches in documentation instead
|
||||
of implementation, invalid source ranges, and failure to recover from tool errors.
|
||||
|
||||
A diagnostic over all successful intermediate reads found target-body coverage
|
||||
on 1/30 tasks for Q4, 0/30 for NF4 and adapter v1, and 1/30 for adapter v2. These
|
||||
are not final-return scores and can consume more context than the returned
|
||||
snippet budget. They show that the problem starts before final selection: most
|
||||
searches never read the target implementation. See [`read-coverage.json`](read-coverage.json).
|
||||
|
||||
## Artifact integrity and diagnostics
|
||||
|
||||
- Base revision: `87179e5c1f455ef22e6223592d2d61351b525bfc`.
|
||||
- Original safetensors file: 2,161,290,912 bytes; SHA-256
|
||||
`7ab8fd86563125929be78aeec8cb3969c7ed2ead3be1ab9d3ec0a9fa69c8660d`.
|
||||
- Official Q4_K_M GGUF revision: `3d55fac80935ae6456986ad2384b5cbcc4d6c948`.
|
||||
- GGUF file: 688,065,920 bytes; SHA-256
|
||||
`81b64d05a23b17b34c475f42b3e72fbde62d4b92cc34541f7a8031d0752deafa`.
|
||||
- Suite SHA-256: `d06effdf41ee2e38bf8f44629949d9b4491067986fc5a73aef22cf80c5a5bf96`.
|
||||
|
||||
Both model-file hashes were verified locally against the publisher's artifacts.
|
||||
The pinned tokenizer and Ollama agreed on the token count of an actual 611-token
|
||||
prompt. A separate JSON-decoding development query, excluded from the 30-task
|
||||
suite, also failed within six rounds with BF16 reference inference (51.18 s).
|
||||
A one-query thinking-mode diagnostic failed too (18.50 s); it is not a benchmark
|
||||
of thinking mode. These checks do not establish that every prompt or runtime
|
||||
configuration would perform equally poorly.
|
||||
|
||||
An early Transformers diagnostic mistakenly removed special XML delimiter tokens
|
||||
during decoding. It was invalidated, corrected, and covered by a regression test.
|
||||
The published Q4 result used Ollama and was unaffected. Only corrected Transformers
|
||||
runs should be used for model comparisons.
|
||||
|
||||
## MCP integration check
|
||||
|
||||
A real stdio client initialized `serve-live` with adapter v2, discovered
|
||||
`scout_live_search`, and made two calls to the same resident server. Both returned
|
||||
valid structured responses without MCP transport errors and with
|
||||
`index_required: false`. Search took 31.89 s and 30.79 s. Both exhausted their
|
||||
six-round budgets on the excluded JSON-decoding development query and returned
|
||||
no source ranges. This validates resident inference and the MCP transport, not
|
||||
successful retrieval. The compact record is [`mcp-smoke.json`](mcp-smoke.json).
|
||||
|
||||
## Reproducibility and limits
|
||||
|
||||
Implementation validation passed: **73 tests**, Ruff lint and format checks,
|
||||
lockfile validation, and source/wheel builds. Tests cover bounded filesystem
|
||||
actions, source verification, protocol decoding, loss masking, and an MCP stdio
|
||||
round trip. These checks validate implementation behavior, not model search quality.
|
||||
|
||||
See [setup and commands](../../docs/LIVE_SEARCH.md). Public JSON artifacts contain
|
||||
configuration, aggregate results, and per-task scores without full source snippets
|
||||
or model-generated reasoning. Full traces, source snapshots, weights, and prepared
|
||||
training data remain in ignored local directories.
|
||||
|
||||
This is a development suite on mature public Python repositories, which may appear
|
||||
in the base model's pretraining. It is not an untouched final holdout. Thirty tasks
|
||||
do not establish performance across repositories, languages, or coding-agent
|
||||
workloads. A gain on this suite would need confirmation on new tasks and a paired
|
||||
solver-with/without-scout experiment before deployment.
|
||||
|
||||
## Recommended next experiment
|
||||
|
||||
Keep this version experimental. First move range arithmetic into the harness:
|
||||
let a policy choose an observed match or span handle, and let the executor expand
|
||||
and validate the corresponding source window. Then train on executed search
|
||||
trajectories in realistic repositories, including unsuccessful searches,
|
||||
reformulations, and distractors. The current short oracle demonstrations mainly
|
||||
teach how to issue actions.
|
||||
|
||||
Freeze a new repository-disjoint evaluation before tuning further. Measure target
|
||||
coverage, returned context size, and latency against lexical and indexed controls.
|
||||
Only then test a larger solver with and without the scout on the same coding
|
||||
tasks. More adapter epochs, dynamic experts, or online reward updates are not
|
||||
supported as the next priority by these results.
|
||||
@@ -0,0 +1,16 @@
|
||||
{
|
||||
"note": "Adapter base-reference metadata normalized after training; numerical weights unchanged. Future trainer saves these fields directly. Exact training source is preserved in source-snapshot.zip.",
|
||||
"base_id": "openbmb/MiniCPM5-1B",
|
||||
"base_revision": "87179e5c1f455ef22e6223592d2d61351b525bfc",
|
||||
"selected_step": 100,
|
||||
"files": {
|
||||
"best/adapter_config.json": {
|
||||
"sha256": "a814d933ef0d056aa4cc162fe949c5dddf12511c9a1551cac06cd8358b1dbab7",
|
||||
"bytes": 970
|
||||
},
|
||||
"best/adapter_model.safetensors": {
|
||||
"sha256": "4a2a5fdcfcd282da485d31e44b552fc4184a71b75073130f62fdfe4af42f34ce",
|
||||
"bytes": 44871152
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,490 @@
|
||||
{
|
||||
"suite": {
|
||||
"name": "live-search-v1",
|
||||
"scope": "30 hand-authored English single-function localization tasks in three public Python repositories. Development benchmark, not a downstream coding-task or contamination-free evaluation.",
|
||||
"repositories": {
|
||||
"requests": {
|
||||
"commit": "b25c87d7cb8d6a18a37fa12442b5f883f9e41741",
|
||||
"url": "https://github.com/psf/requests.git"
|
||||
},
|
||||
"flask": {
|
||||
"commit": "2c1b30d0503cfb064f1cb252e6614a06915a362a",
|
||||
"url": "https://github.com/pallets/flask.git"
|
||||
},
|
||||
"click": {
|
||||
"commit": "fd183b2ced1cb5857784fe7fb22f4982f671f098",
|
||||
"url": "https://github.com/pallets/click.git"
|
||||
}
|
||||
},
|
||||
"tasks": [
|
||||
{
|
||||
"id": "requests-01",
|
||||
"repository": "requests",
|
||||
"query": "Find where an unsuccessful HTTP status becomes an exception containing the server reason and URL.",
|
||||
"targets": [
|
||||
{
|
||||
"path": "src/requests/models.py",
|
||||
"symbol": "Response.raise_for_status",
|
||||
"start_line": 1002,
|
||||
"end_line": 1026,
|
||||
"sha256": "32365d67893bb67c3ed67cf93ca4a18e63e6ab29342fa0dc8b09c59e06ff564e"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "requests-02",
|
||||
"repository": "requests",
|
||||
"query": "Locate the response iterator that preserves an incomplete trailing line across downloaded chunks.",
|
||||
"targets": [
|
||||
{
|
||||
"path": "src/requests/models.py",
|
||||
"symbol": "Response.iter_lines",
|
||||
"start_line": 867,
|
||||
"end_line": 888,
|
||||
"sha256": "32365d67893bb67c3ed67cf93ca4a18e63e6ab29342fa0dc8b09c59e06ff564e"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "requests-03",
|
||||
"repository": "requests",
|
||||
"query": "Where is it decided whether credentials may survive a redirect to another hostname, port, or protocol?",
|
||||
"targets": [
|
||||
{
|
||||
"path": "src/requests/sessions.py",
|
||||
"symbol": "SessionRedirectMixin.should_strip_auth",
|
||||
"start_line": 129,
|
||||
"end_line": 157,
|
||||
"sha256": "0a5d5da449ce7f0af3ccf6e4bbe7a67a935e37846dff4ff9f08cb6c7e2464e6f"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "requests-04",
|
||||
"repository": "requests",
|
||||
"query": "Find the logic that changes the HTTP verb when following 301, 302, or 303 redirects.",
|
||||
"targets": [
|
||||
{
|
||||
"path": "src/requests/sessions.py",
|
||||
"symbol": "SessionRedirectMixin.rebuild_method",
|
||||
"start_line": 337,
|
||||
"end_line": 353,
|
||||
"sha256": "0a5d5da449ce7f0af3ccf6e4bbe7a67a935e37846dff4ff9f08cb6c7e2464e6f"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "requests-05",
|
||||
"repository": "requests",
|
||||
"query": "Locate where environment proxy settings and certificate bundle variables are merged with session options.",
|
||||
"targets": [
|
||||
{
|
||||
"path": "src/requests/sessions.py",
|
||||
"symbol": "Session.merge_environment_settings",
|
||||
"start_line": 757,
|
||||
"end_line": 779,
|
||||
"sha256": "0a5d5da449ce7f0af3ccf6e4bbe7a67a935e37846dff4ff9f08cb6c7e2464e6f"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "requests-06",
|
||||
"repository": "requests",
|
||||
"query": "Find where a 401 digest challenge causes the original request to be copied and sent again with authentication.",
|
||||
"targets": [
|
||||
{
|
||||
"path": "src/requests/auth.py",
|
||||
"symbol": "HTTPDigestAuth.handle_401",
|
||||
"start_line": 250,
|
||||
"end_line": 283,
|
||||
"sha256": "905ef9b6a9cb72d67d31ffe19bd4d9223e1c4169cde6ec51cfca16b31e70991d"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "requests-07",
|
||||
"repository": "requests",
|
||||
"query": "Locate the helper that fills a cookie container from a mapping while optionally preserving existing names.",
|
||||
"targets": [
|
||||
{
|
||||
"path": "src/requests/cookies.py",
|
||||
"symbol": "cookiejar_from_dict",
|
||||
"start_line": 530,
|
||||
"end_line": 539,
|
||||
"sha256": "6cd8be8aa123e0d3d9d34fa86feac7bf392f39bccdde5129830de0ea9692dd7c"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "requests-08",
|
||||
"repository": "requests",
|
||||
"query": "Find where credentials are loaded from the users netrc file, including NETRC and home-directory lookup.",
|
||||
"targets": [
|
||||
{
|
||||
"path": "src/requests/utils.py",
|
||||
"symbol": "get_netrc_auth",
|
||||
"start_line": 210,
|
||||
"end_line": 248,
|
||||
"sha256": "5aa53ceab677c2f842fad42359c8ed1ff1c4299c1607789609957a496e4311d4"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "requests-09",
|
||||
"repository": "requests",
|
||||
"query": "Find the implementation that seeks a request body back to its saved position and fails for an unrewindable stream.",
|
||||
"targets": [
|
||||
{
|
||||
"path": "src/requests/utils.py",
|
||||
"symbol": "rewind_body",
|
||||
"start_line": 1075,
|
||||
"end_line": 1086,
|
||||
"sha256": "5aa53ceab677c2f842fad42359c8ed1ff1c4299c1607789609957a496e4311d4"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "requests-10",
|
||||
"repository": "requests",
|
||||
"query": "Where is a request URL validated, its international hostname encoded, and its query parameters appended?",
|
||||
"targets": [
|
||||
{
|
||||
"path": "src/requests/models.py",
|
||||
"symbol": "PreparedRequest.prepare_url",
|
||||
"start_line": 416,
|
||||
"end_line": 481,
|
||||
"sha256": "32365d67893bb67c3ed67cf93ca4a18e63e6ab29342fa0dc8b09c59e06ff564e"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "flask-01",
|
||||
"repository": "flask",
|
||||
"query": "Find the code that invokes the view function selected by the matched URL rule.",
|
||||
"targets": [
|
||||
{
|
||||
"path": "src/flask/app.py",
|
||||
"symbol": "Flask.dispatch_request",
|
||||
"start_line": 889,
|
||||
"end_line": 902,
|
||||
"sha256": "5c6aa0151b0b8018732280761d27eda0c4c83378630e21faadd57b73783720a7"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "flask-02",
|
||||
"repository": "flask",
|
||||
"query": "Where are view return values such as tuples, dictionaries and strings converted into a response object?",
|
||||
"targets": [
|
||||
{
|
||||
"path": "src/flask/app.py",
|
||||
"symbol": "Flask.make_response",
|
||||
"start_line": 1186,
|
||||
"end_line": 1269,
|
||||
"sha256": "5c6aa0151b0b8018732280761d27eda0c4c83378630e21faadd57b73783720a7"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "flask-03",
|
||||
"repository": "flask",
|
||||
"query": "Find where coroutine view functions are adapted for synchronous request handling.",
|
||||
"targets": [
|
||||
{
|
||||
"path": "src/flask/app.py",
|
||||
"symbol": "Flask.ensure_sync",
|
||||
"start_line": 975,
|
||||
"end_line": 978,
|
||||
"sha256": "5c6aa0151b0b8018732280761d27eda0c4c83378630e21faadd57b73783720a7"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "flask-04",
|
||||
"repository": "flask",
|
||||
"query": "Locate the processing that runs after-request callbacks and saves the session before returning the response.",
|
||||
"targets": [
|
||||
{
|
||||
"path": "src/flask/app.py",
|
||||
"symbol": "Flask.process_response",
|
||||
"start_line": 1311,
|
||||
"end_line": 1324,
|
||||
"sha256": "5c6aa0151b0b8018732280761d27eda0c4c83378630e21faadd57b73783720a7"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "flask-05",
|
||||
"repository": "flask",
|
||||
"query": "Find where before-request handlers can short-circuit normal request dispatch by returning a value.",
|
||||
"targets": [
|
||||
{
|
||||
"path": "src/flask/app.py",
|
||||
"symbol": "Flask.preprocess_request",
|
||||
"start_line": 1281,
|
||||
"end_line": 1296,
|
||||
"sha256": "5c6aa0151b0b8018732280761d27eda0c4c83378630e21faadd57b73783720a7"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "flask-06",
|
||||
"repository": "flask",
|
||||
"query": "Where is a streaming generator wrapped so the request context remains active while producing its items?",
|
||||
"targets": [
|
||||
{
|
||||
"path": "src/flask/helpers.py",
|
||||
"symbol": "stream_with_context",
|
||||
"start_line": 104,
|
||||
"end_line": 143,
|
||||
"sha256": "ac96607bbfc9dbcf09d5442fe7e90d7f8a046b0df7d8f3fc3535b44112016d71"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "flask-07",
|
||||
"repository": "flask",
|
||||
"query": "Find where a categorized one-time message is stored in the session and a notification signal is emitted.",
|
||||
"targets": [
|
||||
{
|
||||
"path": "src/flask/helpers.py",
|
||||
"symbol": "flash",
|
||||
"start_line": 340,
|
||||
"end_line": 349,
|
||||
"sha256": "ac96607bbfc9dbcf09d5442fe7e90d7f8a046b0df7d8f3fc3535b44112016d71"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "flask-08",
|
||||
"repository": "flask",
|
||||
"query": "Locate where stored one-time messages are popped from the session, cached for the request, and filtered by category.",
|
||||
"targets": [
|
||||
{
|
||||
"path": "src/flask/helpers.py",
|
||||
"symbol": "get_flashed_messages",
|
||||
"start_line": 383,
|
||||
"end_line": 391,
|
||||
"sha256": "ac96607bbfc9dbcf09d5442fe7e90d7f8a046b0df7d8f3fc3535b44112016d71"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "flask-09",
|
||||
"repository": "flask",
|
||||
"query": "Find where the command-line startup reads dotenv files without replacing environment variables already set.",
|
||||
"targets": [
|
||||
{
|
||||
"path": "src/flask/cli.py",
|
||||
"symbol": "load_dotenv",
|
||||
"start_line": 740,
|
||||
"end_line": 771,
|
||||
"sha256": "3df87bdbe07196fa07d101c20ce7351ef5c6ecaab95deb5fdf3ccdcf690d4879"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "flask-10",
|
||||
"repository": "flask",
|
||||
"query": "Locate where a signed session cookie is verified and an invalid signature produces an empty session.",
|
||||
"targets": [
|
||||
{
|
||||
"path": "src/flask/sessions.py",
|
||||
"symbol": "SecureCookieSessionInterface.open_session",
|
||||
"start_line": 338,
|
||||
"end_line": 349,
|
||||
"sha256": "76ebd81a608687f1f772032ecb6a1e4a3ac2b02bcbca517808128e5e9374426b"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "click-01",
|
||||
"repository": "click",
|
||||
"query": "Find where a supplied option value is normalized and matched against the allowed choices.",
|
||||
"targets": [
|
||||
{
|
||||
"path": "src/click/types.py",
|
||||
"symbol": "Choice.convert",
|
||||
"start_line": 344,
|
||||
"end_line": 358,
|
||||
"sha256": "2814d1c4ddbc711d5567999bf62257f7c3104b0fe9f521b3963a9f108f33e53c"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "click-02",
|
||||
"repository": "click",
|
||||
"query": "Locate where a filesystem argument is checked for existence, readability, writability, and allowed file or directory type.",
|
||||
"targets": [
|
||||
{
|
||||
"path": "src/click/types.py",
|
||||
"symbol": "Path.convert",
|
||||
"start_line": 930,
|
||||
"end_line": 995,
|
||||
"sha256": "2814d1c4ddbc711d5567999bf62257f7c3104b0fe9f521b3963a9f108f33e53c"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "click-03",
|
||||
"repository": "click",
|
||||
"query": "Find where a date argument is parsed by trying several accepted formats and reports a failure if none match.",
|
||||
"targets": [
|
||||
{
|
||||
"path": "src/click/types.py",
|
||||
"symbol": "DateTime.convert",
|
||||
"start_line": 448,
|
||||
"end_line": 466,
|
||||
"sha256": "2814d1c4ddbc711d5567999bf62257f7c3104b0fe9f521b3963a9f108f33e53c"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "click-04",
|
||||
"repository": "click",
|
||||
"query": "Where is an option value from an environment variable split and grouped for multiple arguments?",
|
||||
"targets": [
|
||||
{
|
||||
"path": "src/click/core.py",
|
||||
"symbol": "Option.value_from_envvar",
|
||||
"start_line": 2983,
|
||||
"end_line": 2996,
|
||||
"sha256": "814869352f5c14119d1175dd8ac1951be791bc67f8f514f215a831ba5ab0745c"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "click-05",
|
||||
"repository": "click",
|
||||
"query": "Locate where a command group resolves a subcommand name, retries normalized names, and rejects unknown commands.",
|
||||
"targets": [
|
||||
{
|
||||
"path": "src/click/core.py",
|
||||
"symbol": "Group.resolve_command",
|
||||
"start_line": 1867,
|
||||
"end_line": 1889,
|
||||
"sha256": "814869352f5c14119d1175dd8ac1951be791bc67f8f514f215a831ba5ab0745c"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "click-06",
|
||||
"repository": "click",
|
||||
"query": "Find the helper that enters a context manager and registers its cleanup with the command context.",
|
||||
"targets": [
|
||||
{
|
||||
"path": "src/click/core.py",
|
||||
"symbol": "Context.with_resource",
|
||||
"start_line": 598,
|
||||
"end_line": 598,
|
||||
"sha256": "814869352f5c14119d1175dd8ac1951be791bc67f8f514f215a831ba5ab0745c"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "click-07",
|
||||
"repository": "click",
|
||||
"query": "Find where each element of a tuple argument is converted using its corresponding parameter type.",
|
||||
"targets": [
|
||||
{
|
||||
"path": "src/click/types.py",
|
||||
"symbol": "Tuple.convert",
|
||||
"start_line": 1049,
|
||||
"end_line": 1065,
|
||||
"sha256": "2814d1c4ddbc711d5567999bf62257f7c3104b0fe9f521b3963a9f108f33e53c"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "click-08",
|
||||
"repository": "click",
|
||||
"query": "Locate the interactive yes-or-no question loop that retries invalid answers and can abort on rejection.",
|
||||
"targets": [
|
||||
{
|
||||
"path": "src/click/termui.py",
|
||||
"symbol": "confirm",
|
||||
"start_line": 223,
|
||||
"end_line": 252,
|
||||
"sha256": "bc062b282d9aedffcd7c42210138445587dafff89be5741b4d2086b499400510"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "click-09",
|
||||
"repository": "click",
|
||||
"query": "Find the testing helper that temporarily changes the current directory and removes its temporary directory on exit.",
|
||||
"targets": [
|
||||
{
|
||||
"path": "src/click/testing.py",
|
||||
"symbol": "CliRunner.isolated_filesystem",
|
||||
"start_line": 552,
|
||||
"end_line": 565,
|
||||
"sha256": "d9e2dd01a0890864e83f94f0f9737c263ac36f7c1b2487c047183777472b5b93"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "click-10",
|
||||
"repository": "click",
|
||||
"query": "Where does a lazily opened file acquire its actual stream and translate operating-system errors into a file error?",
|
||||
"targets": [
|
||||
{
|
||||
"path": "src/click/utils.py",
|
||||
"symbol": "LazyFile.open",
|
||||
"start_line": 156,
|
||||
"end_line": 167,
|
||||
"sha256": "6f5326faeb040c11ed13070f96d3c89f7cd22b89f0d8a4e9e460bbfe849394ba"
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
},
|
||||
"suite_sha256": "d06effdf41ee2e38bf8f44629949d9b4491067986fc5a73aef22cf80c5a5bf96",
|
||||
"model": {
|
||||
"model": "openbmb/MiniCPM5-1B+best",
|
||||
"base_id": "openbmb/MiniCPM5-1B",
|
||||
"base_revision": "87179e5c1f455ef22e6223592d2d61351b525bfc",
|
||||
"backend": "transformers",
|
||||
"quantization": "nf4",
|
||||
"adapter": "runs/minicpm5-policy-v1/best",
|
||||
"load_seconds": 15.25091333999444
|
||||
},
|
||||
"source_sha256": {
|
||||
"__init__.py": "dcd2b573883b8068e806e3052adf9c03728f1cf621d35ff013179de76f76c702",
|
||||
"text.py": "8db7b80ee446480175f7691872303c8b72fd014604ad4fb575b359cee71f1938",
|
||||
"io.py": "b29c21db767ea9773281b17fe0754375153d18fb64f314907528f00d584a047f",
|
||||
"data.py": "20a891eff11d555d2301b62f4953146603efc87b03da4802f136f694ebdba78d",
|
||||
"encoder.py": "cb5671c08f28a70ffd0e2a7033528efc3581896a27242d55eb250ce922ab038f",
|
||||
"metrics.py": "cb93d30878c0c1006f06d1d0c4cbb188f90c6f4fc65f69f9a3b11d847f28ad27",
|
||||
"train.py": "cf20f6189967cfc88f3849bc3a225f2680c42026d53f204d9d5afc05fe88a2d6",
|
||||
"lexical.py": "6c47c834c6bbf7b317e2514057eee7444720581f5ec452ecb01ac666397b86b3",
|
||||
"symbols.py": "a377d6aa2c6b479c2ee115dd8e4859f0c06d188ae97b86fa1c7133885ee32d86",
|
||||
"index.py": "e8b0a666b41fed48d98616d5dcdd104e59c4308ac990281b463e5ee68f97877f",
|
||||
"evaluate.py": "29c942b9cbfff21ba8514511e27752527f0ec5e2cad5fbf57ec3b220fc7acd0b",
|
||||
"__main__.py": "ab30d9b696e41ece67094e19c87b1a31fc0ebce8d3afa79dc8eaf561f8fb4895",
|
||||
"download.py": "24f9802fb7cc82b81f6b1cb93e9244352d7b86feeb0a54ec055fb3e8d0a6a4d7",
|
||||
"cli.py": "d081962fc459808c65bd4c1d389e12ad9a9466df0a1374c8095a8bd5ccbb369e",
|
||||
"scout.py": "c3150baae51f37646305fee1d2c61e7b37f2ec9c4a0ba7f92ab4d7d6b1710161",
|
||||
"server.py": "483fe8051012249e38a6836cbe217d8c581d033e4d19026a29a7bd5c83878190",
|
||||
"live_tools.py": "e10fb6aaeaa6d7c4a5a35c0c6a210be4a4c283b18d215767d6bee0bac707d14c",
|
||||
"local_policy.py": "b0711b46a6399dfbeeecdcc045fb7392030e187d7e21c6e4fa9b51e6fc8056f4",
|
||||
"agent.py": "47357a900e9696b7ec5268eb031067eb0069faaeef92fd3159bd822d6bc67cdb",
|
||||
"live_server.py": "ccfb3c63115fe991c2f5f22c05f794d89c90d783b52f8ddbb668a8541e3af6fe",
|
||||
"native_protocol.py": "3a4d16c1d2cd90cb7d81e635f40d721270b0d541f454557a6d14e80b6b1073a7",
|
||||
"eval_live.py": "38250d45fd7e60c23c4c86f5819242736f29df8af384f92713128c84ffd734ff",
|
||||
"prepare_live.py": "0675173c5e9c116c995c271dc06de914388f19eb83779c07e3009000a810290b",
|
||||
"live_data.py": "fd0bd5b118b2437146e08ef921dc4aa8a1da24e69bcbdd3efc5ff6e9c4fe8a78",
|
||||
"transformers_policy.py": "e8e8a81b70eb37563973e640534b425a7ea9fc7d398a39dc8d8f71b9be3f1b89",
|
||||
"train_policy.py": "785760ddb1373cc7b37b8e0bd4afa02560d8ebb512cf401f932bbca541f1e930",
|
||||
"keyword_baseline.py": "5fe18c76b4c6e085c358ab5c490d717ffdd82b917a2882cf826fa26c57f64816"
|
||||
},
|
||||
"max_rounds": 6,
|
||||
"max_chars": 6000,
|
||||
"timeout_seconds": 90,
|
||||
"context": 8192,
|
||||
"max_generation_tokens": 512,
|
||||
"tokenizer": "pinned_hf",
|
||||
"temperature": 0,
|
||||
"seed": 42,
|
||||
"scope": "Single-function localization; no large-model baseline; public repositories may have appeared in the base model's pretraining."
|
||||
}
|
||||
@@ -0,0 +1,87 @@
|
||||
{
|
||||
"tasks": 30,
|
||||
"target_hit_rate": 0,
|
||||
"file_hit_rate": 0.26666666666666666,
|
||||
"macro_line_precision": 0.0,
|
||||
"macro_line_recall": 0.0,
|
||||
"macro_line_f1": 0.0,
|
||||
"latency_median_seconds": 19.395455925499846,
|
||||
"latency_p95_seconds": 39.98146147420266,
|
||||
"statuses": {
|
||||
"completed": 25,
|
||||
"budget_exhausted": 5
|
||||
},
|
||||
"total_tool_errors": 14,
|
||||
"total_invalid_actions": 13,
|
||||
"mean_rounds": 3.8666666666666667,
|
||||
"mean_tool_calls": 3.6,
|
||||
"total_input_tokens": 304404,
|
||||
"total_output_tokens": 3579,
|
||||
"mean_returned_chars": 1342.1333333333334,
|
||||
"per_repository": {
|
||||
"requests": {
|
||||
"tasks": 10,
|
||||
"target_hit_rate": 0,
|
||||
"file_hit_rate": 0.5,
|
||||
"macro_line_precision": 0.0,
|
||||
"macro_line_recall": 0.0,
|
||||
"macro_line_f1": 0.0,
|
||||
"latency_median_seconds": 18.327982628001337,
|
||||
"latency_p95_seconds": 34.40433353035005,
|
||||
"statuses": {
|
||||
"completed": 8,
|
||||
"budget_exhausted": 2
|
||||
},
|
||||
"total_tool_errors": 3,
|
||||
"total_invalid_actions": 4,
|
||||
"mean_rounds": 3.6,
|
||||
"mean_tool_calls": 3.4,
|
||||
"total_input_tokens": 95640,
|
||||
"total_output_tokens": 1089,
|
||||
"mean_returned_chars": 953.9
|
||||
},
|
||||
"flask": {
|
||||
"tasks": 10,
|
||||
"target_hit_rate": 0,
|
||||
"file_hit_rate": 0.2,
|
||||
"macro_line_precision": 0.0,
|
||||
"macro_line_recall": 0.0,
|
||||
"macro_line_f1": 0.0,
|
||||
"latency_median_seconds": 19.156648668500566,
|
||||
"latency_p95_seconds": 38.20192200995334,
|
||||
"statuses": {
|
||||
"completed": 7,
|
||||
"budget_exhausted": 3
|
||||
},
|
||||
"total_tool_errors": 7,
|
||||
"total_invalid_actions": 6,
|
||||
"mean_rounds": 4.2,
|
||||
"mean_tool_calls": 3.9,
|
||||
"total_input_tokens": 109394,
|
||||
"total_output_tokens": 1297,
|
||||
"mean_returned_chars": 1286
|
||||
},
|
||||
"click": {
|
||||
"tasks": 10,
|
||||
"target_hit_rate": 0,
|
||||
"file_hit_rate": 0.1,
|
||||
"macro_line_precision": 0.0,
|
||||
"macro_line_recall": 0.0,
|
||||
"macro_line_f1": 0.0,
|
||||
"latency_median_seconds": 21.76824945649787,
|
||||
"latency_p95_seconds": 42.4032184362477,
|
||||
"statuses": {
|
||||
"completed": 10
|
||||
},
|
||||
"total_tool_errors": 4,
|
||||
"total_invalid_actions": 3,
|
||||
"mean_rounds": 3.8,
|
||||
"mean_tool_calls": 3.5,
|
||||
"total_input_tokens": 99370,
|
||||
"total_output_tokens": 1193,
|
||||
"mean_returned_chars": 1786.5
|
||||
}
|
||||
},
|
||||
"sampled_peak_gpu_memory_mib": 2376,
|
||||
"gpu_sampling_interval_seconds": 1
|
||||
}
|
||||
@@ -0,0 +1,892 @@
|
||||
[
|
||||
{
|
||||
"id": "requests-01",
|
||||
"repository": "requests",
|
||||
"scores": {
|
||||
"file_hit": false,
|
||||
"target_hit": false,
|
||||
"line_precision": 0.0,
|
||||
"line_recall": 0.0,
|
||||
"line_f1": 0.0,
|
||||
"returned_lines": 40,
|
||||
"overlap_lines": 0
|
||||
},
|
||||
"status": "completed",
|
||||
"elapsed_seconds": 18.666427345000557,
|
||||
"rounds": 3,
|
||||
"tool_calls": 3,
|
||||
"tool_errors": 0,
|
||||
"invalid_actions": 0,
|
||||
"input_tokens": 7871,
|
||||
"output_tokens": 93,
|
||||
"returned_chars": 1055,
|
||||
"references": [
|
||||
{
|
||||
"path": "src/requests/exceptions.py",
|
||||
"start_line": 81,
|
||||
"end_line": 120,
|
||||
"sha256": "8c93d2d545804ecf3a4a155468ba2b4e225bd52686ba83445a020225ea7e5646",
|
||||
"verified": true
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "requests-02",
|
||||
"repository": "requests",
|
||||
"scores": {
|
||||
"file_hit": false,
|
||||
"target_hit": false,
|
||||
"line_precision": 0.0,
|
||||
"line_recall": 0.0,
|
||||
"line_f1": 0.0,
|
||||
"returned_lines": 35,
|
||||
"overlap_lines": 0
|
||||
},
|
||||
"status": "completed",
|
||||
"elapsed_seconds": 22.915221289003966,
|
||||
"rounds": 3,
|
||||
"tool_calls": 3,
|
||||
"tool_errors": 0,
|
||||
"invalid_actions": 0,
|
||||
"input_tokens": 6303,
|
||||
"output_tokens": 93,
|
||||
"returned_chars": 1000,
|
||||
"references": [
|
||||
{
|
||||
"path": "src/requests/cookies.py",
|
||||
"start_line": 226,
|
||||
"end_line": 260,
|
||||
"sha256": "6cd8be8aa123e0d3d9d34fa86feac7bf392f39bccdde5129830de0ea9692dd7c",
|
||||
"verified": true
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "requests-03",
|
||||
"repository": "requests",
|
||||
"scores": {
|
||||
"file_hit": true,
|
||||
"target_hit": false,
|
||||
"line_precision": 0.0,
|
||||
"line_recall": 0.0,
|
||||
"line_f1": 0.0,
|
||||
"returned_lines": 1,
|
||||
"overlap_lines": 0
|
||||
},
|
||||
"status": "completed",
|
||||
"elapsed_seconds": 23.441803863999667,
|
||||
"rounds": 3,
|
||||
"tool_calls": 3,
|
||||
"tool_errors": 0,
|
||||
"invalid_actions": 0,
|
||||
"input_tokens": 6314,
|
||||
"output_tokens": 92,
|
||||
"returned_chars": 79,
|
||||
"references": [
|
||||
{
|
||||
"path": "src/requests/sessions.py",
|
||||
"start_line": 284,
|
||||
"end_line": 284,
|
||||
"sha256": "0a5d5da449ce7f0af3ccf6e4bbe7a67a935e37846dff4ff9f08cb6c7e2464e6f",
|
||||
"verified": true
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "requests-04",
|
||||
"repository": "requests",
|
||||
"scores": {
|
||||
"file_hit": true,
|
||||
"target_hit": false,
|
||||
"line_precision": 0.0,
|
||||
"line_recall": 0.0,
|
||||
"line_f1": 0.0,
|
||||
"returned_lines": 84,
|
||||
"overlap_lines": 0
|
||||
},
|
||||
"status": "completed",
|
||||
"elapsed_seconds": 17.989537911002117,
|
||||
"rounds": 3,
|
||||
"tool_calls": 3,
|
||||
"tool_errors": 0,
|
||||
"invalid_actions": 0,
|
||||
"input_tokens": 8137,
|
||||
"output_tokens": 91,
|
||||
"returned_chars": 2675,
|
||||
"references": [
|
||||
{
|
||||
"path": "src/requests/sessions.py",
|
||||
"start_line": 28,
|
||||
"end_line": 111,
|
||||
"sha256": "0a5d5da449ce7f0af3ccf6e4bbe7a67a935e37846dff4ff9f08cb6c7e2464e6f",
|
||||
"verified": true
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "requests-05",
|
||||
"repository": "requests",
|
||||
"scores": {
|
||||
"file_hit": false,
|
||||
"target_hit": false,
|
||||
"line_precision": 0.0,
|
||||
"line_recall": 0.0,
|
||||
"line_f1": 0.0,
|
||||
"returned_lines": 0,
|
||||
"overlap_lines": 0
|
||||
},
|
||||
"status": "budget_exhausted",
|
||||
"elapsed_seconds": 33.494351162000385,
|
||||
"rounds": 6,
|
||||
"tool_calls": 5,
|
||||
"tool_errors": 1,
|
||||
"invalid_actions": 2,
|
||||
"input_tokens": 19536,
|
||||
"output_tokens": 171,
|
||||
"returned_chars": 0,
|
||||
"references": []
|
||||
},
|
||||
{
|
||||
"id": "requests-06",
|
||||
"repository": "requests",
|
||||
"scores": {
|
||||
"file_hit": true,
|
||||
"target_hit": false,
|
||||
"line_precision": 0.0,
|
||||
"line_recall": 0.0,
|
||||
"line_f1": 0.0,
|
||||
"returned_lines": 66,
|
||||
"overlap_lines": 0
|
||||
},
|
||||
"status": "completed",
|
||||
"elapsed_seconds": 17.204540024999005,
|
||||
"rounds": 3,
|
||||
"tool_calls": 3,
|
||||
"tool_errors": 0,
|
||||
"invalid_actions": 0,
|
||||
"input_tokens": 8087,
|
||||
"output_tokens": 92,
|
||||
"returned_chars": 2237,
|
||||
"references": [
|
||||
{
|
||||
"path": "src/requests/auth.py",
|
||||
"start_line": 107,
|
||||
"end_line": 172,
|
||||
"sha256": "905ef9b6a9cb72d67d31ffe19bd4d9223e1c4169cde6ec51cfca16b31e70991d",
|
||||
"verified": true
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "requests-07",
|
||||
"repository": "requests",
|
||||
"scores": {
|
||||
"file_hit": true,
|
||||
"target_hit": false,
|
||||
"line_precision": 0.0,
|
||||
"line_recall": 0.0,
|
||||
"line_f1": 0.0,
|
||||
"returned_lines": 78,
|
||||
"overlap_lines": 0
|
||||
},
|
||||
"status": "completed",
|
||||
"elapsed_seconds": 17.731084020000708,
|
||||
"rounds": 3,
|
||||
"tool_calls": 3,
|
||||
"tool_errors": 0,
|
||||
"invalid_actions": 0,
|
||||
"input_tokens": 8086,
|
||||
"output_tokens": 94,
|
||||
"returned_chars": 2373,
|
||||
"references": [
|
||||
{
|
||||
"path": "src/requests/cookies.py",
|
||||
"start_line": 2,
|
||||
"end_line": 79,
|
||||
"sha256": "6cd8be8aa123e0d3d9d34fa86feac7bf392f39bccdde5129830de0ea9692dd7c",
|
||||
"verified": true
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "requests-08",
|
||||
"repository": "requests",
|
||||
"scores": {
|
||||
"file_hit": false,
|
||||
"target_hit": false,
|
||||
"line_precision": 0.0,
|
||||
"line_recall": 0.0,
|
||||
"line_f1": 0.0,
|
||||
"returned_lines": 1,
|
||||
"overlap_lines": 0
|
||||
},
|
||||
"status": "completed",
|
||||
"elapsed_seconds": 16.809887543000514,
|
||||
"rounds": 3,
|
||||
"tool_calls": 3,
|
||||
"tool_errors": 0,
|
||||
"invalid_actions": 0,
|
||||
"input_tokens": 6317,
|
||||
"output_tokens": 92,
|
||||
"returned_chars": 79,
|
||||
"references": [
|
||||
{
|
||||
"path": "src/requests/sessions.py",
|
||||
"start_line": 284,
|
||||
"end_line": 284,
|
||||
"sha256": "0a5d5da449ce7f0af3ccf6e4bbe7a67a935e37846dff4ff9f08cb6c7e2464e6f",
|
||||
"verified": true
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "requests-09",
|
||||
"repository": "requests",
|
||||
"scores": {
|
||||
"file_hit": false,
|
||||
"target_hit": false,
|
||||
"line_precision": 0.0,
|
||||
"line_recall": 0.0,
|
||||
"line_f1": 0.0,
|
||||
"returned_lines": 0,
|
||||
"overlap_lines": 0
|
||||
},
|
||||
"status": "budget_exhausted",
|
||||
"elapsed_seconds": 35.148864558999776,
|
||||
"rounds": 6,
|
||||
"tool_calls": 5,
|
||||
"tool_errors": 2,
|
||||
"invalid_actions": 2,
|
||||
"input_tokens": 17766,
|
||||
"output_tokens": 182,
|
||||
"returned_chars": 0,
|
||||
"references": []
|
||||
},
|
||||
{
|
||||
"id": "requests-10",
|
||||
"repository": "requests",
|
||||
"scores": {
|
||||
"file_hit": true,
|
||||
"target_hit": false,
|
||||
"line_precision": 0.0,
|
||||
"line_recall": 0.0,
|
||||
"line_f1": 0.0,
|
||||
"returned_lines": 2,
|
||||
"overlap_lines": 0
|
||||
},
|
||||
"status": "completed",
|
||||
"elapsed_seconds": 16.14751815100317,
|
||||
"rounds": 3,
|
||||
"tool_calls": 3,
|
||||
"tool_errors": 0,
|
||||
"invalid_actions": 0,
|
||||
"input_tokens": 7223,
|
||||
"output_tokens": 89,
|
||||
"returned_chars": 41,
|
||||
"references": [
|
||||
{
|
||||
"path": "src/requests/models.py",
|
||||
"start_line": 99,
|
||||
"end_line": 100,
|
||||
"sha256": "32365d67893bb67c3ed67cf93ca4a18e63e6ab29342fa0dc8b09c59e06ff564e",
|
||||
"verified": true
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "flask-01",
|
||||
"repository": "flask",
|
||||
"scores": {
|
||||
"file_hit": true,
|
||||
"target_hit": false,
|
||||
"line_precision": 0.0,
|
||||
"line_recall": 0.0,
|
||||
"line_f1": 0.0,
|
||||
"returned_lines": 118,
|
||||
"overlap_lines": 0
|
||||
},
|
||||
"status": "completed",
|
||||
"elapsed_seconds": 17.086220979996142,
|
||||
"rounds": 3,
|
||||
"tool_calls": 3,
|
||||
"tool_errors": 0,
|
||||
"invalid_actions": 0,
|
||||
"input_tokens": 8815,
|
||||
"output_tokens": 89,
|
||||
"returned_chars": 4870,
|
||||
"references": [
|
||||
{
|
||||
"path": "src/flask/app.py",
|
||||
"start_line": 12,
|
||||
"end_line": 129,
|
||||
"sha256": "5c6aa0151b0b8018732280761d27eda0c4c83378630e21faadd57b73783720a7",
|
||||
"verified": true
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "flask-02",
|
||||
"repository": "flask",
|
||||
"scores": {
|
||||
"file_hit": false,
|
||||
"target_hit": false,
|
||||
"line_precision": 0.0,
|
||||
"line_recall": 0.0,
|
||||
"line_f1": 0.0,
|
||||
"returned_lines": 23,
|
||||
"overlap_lines": 0
|
||||
},
|
||||
"status": "completed",
|
||||
"elapsed_seconds": 16.1909588540002,
|
||||
"rounds": 3,
|
||||
"tool_calls": 3,
|
||||
"tool_errors": 0,
|
||||
"invalid_actions": 0,
|
||||
"input_tokens": 7491,
|
||||
"output_tokens": 93,
|
||||
"returned_chars": 1316,
|
||||
"references": [
|
||||
{
|
||||
"path": "src/flask/__init__.py",
|
||||
"start_line": 21,
|
||||
"end_line": 43,
|
||||
"sha256": "987bc937d4b0b65d510ed8c2a82218c889e22bf499fe5fe1a94ca73b382927da",
|
||||
"verified": true
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "flask-03",
|
||||
"repository": "flask",
|
||||
"scores": {
|
||||
"file_hit": false,
|
||||
"target_hit": false,
|
||||
"line_precision": 0.0,
|
||||
"line_recall": 0.0,
|
||||
"line_f1": 0.0,
|
||||
"returned_lines": 33,
|
||||
"overlap_lines": 0
|
||||
},
|
||||
"status": "completed",
|
||||
"elapsed_seconds": 34.5737157629992,
|
||||
"rounds": 6,
|
||||
"tool_calls": 5,
|
||||
"tool_errors": 1,
|
||||
"invalid_actions": 1,
|
||||
"input_tokens": 15748,
|
||||
"output_tokens": 187,
|
||||
"returned_chars": 1294,
|
||||
"references": [
|
||||
{
|
||||
"path": "src/flask/sansio/app.py",
|
||||
"start_line": 31,
|
||||
"end_line": 63,
|
||||
"sha256": "e446f1c0739c86072966a4326a503dbf20c1a299273af0e0e815705c5c850f6b",
|
||||
"verified": true
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "flask-04",
|
||||
"repository": "flask",
|
||||
"scores": {
|
||||
"file_hit": false,
|
||||
"target_hit": false,
|
||||
"line_precision": 0.0,
|
||||
"line_recall": 0.0,
|
||||
"line_f1": 0.0,
|
||||
"returned_lines": 0,
|
||||
"overlap_lines": 0
|
||||
},
|
||||
"status": "budget_exhausted",
|
||||
"elapsed_seconds": 37.34333524599788,
|
||||
"rounds": 6,
|
||||
"tool_calls": 5,
|
||||
"tool_errors": 2,
|
||||
"invalid_actions": 2,
|
||||
"input_tokens": 16213,
|
||||
"output_tokens": 196,
|
||||
"returned_chars": 0,
|
||||
"references": []
|
||||
},
|
||||
{
|
||||
"id": "flask-05",
|
||||
"repository": "flask",
|
||||
"scores": {
|
||||
"file_hit": false,
|
||||
"target_hit": false,
|
||||
"line_precision": 0.0,
|
||||
"line_recall": 0.0,
|
||||
"line_f1": 0.0,
|
||||
"returned_lines": 53,
|
||||
"overlap_lines": 0
|
||||
},
|
||||
"status": "completed",
|
||||
"elapsed_seconds": 19.359624311000516,
|
||||
"rounds": 3,
|
||||
"tool_calls": 3,
|
||||
"tool_errors": 0,
|
||||
"invalid_actions": 0,
|
||||
"input_tokens": 7834,
|
||||
"output_tokens": 93,
|
||||
"returned_chars": 2501,
|
||||
"references": [
|
||||
{
|
||||
"path": "src/flask/__init__.py",
|
||||
"start_line": 9,
|
||||
"end_line": 61,
|
||||
"sha256": "987bc937d4b0b65d510ed8c2a82218c889e22bf499fe5fe1a94ca73b382927da",
|
||||
"verified": true
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "flask-06",
|
||||
"repository": "flask",
|
||||
"scores": {
|
||||
"file_hit": false,
|
||||
"target_hit": false,
|
||||
"line_precision": 0.0,
|
||||
"line_recall": 0.0,
|
||||
"line_f1": 0.0,
|
||||
"returned_lines": 17,
|
||||
"overlap_lines": 0
|
||||
},
|
||||
"status": "completed",
|
||||
"elapsed_seconds": 18.953673026000615,
|
||||
"rounds": 3,
|
||||
"tool_calls": 3,
|
||||
"tool_errors": 0,
|
||||
"invalid_actions": 0,
|
||||
"input_tokens": 6463,
|
||||
"output_tokens": 94,
|
||||
"returned_chars": 742,
|
||||
"references": [
|
||||
{
|
||||
"path": "src/flask/templating.py",
|
||||
"start_line": 194,
|
||||
"end_line": 210,
|
||||
"sha256": "207b1db05f9e0493c2244d0024b0a2d558619f2b4e19dcc70a7df24e1cfcedce",
|
||||
"verified": true
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "flask-07",
|
||||
"repository": "flask",
|
||||
"scores": {
|
||||
"file_hit": false,
|
||||
"target_hit": false,
|
||||
"line_precision": 0.0,
|
||||
"line_recall": 0.0,
|
||||
"line_f1": 0.0,
|
||||
"returned_lines": 0,
|
||||
"overlap_lines": 0
|
||||
},
|
||||
"status": "budget_exhausted",
|
||||
"elapsed_seconds": 38.183569715001795,
|
||||
"rounds": 6,
|
||||
"tool_calls": 5,
|
||||
"tool_errors": 2,
|
||||
"invalid_actions": 2,
|
||||
"input_tokens": 16213,
|
||||
"output_tokens": 196,
|
||||
"returned_chars": 0,
|
||||
"references": []
|
||||
},
|
||||
{
|
||||
"id": "flask-08",
|
||||
"repository": "flask",
|
||||
"scores": {
|
||||
"file_hit": true,
|
||||
"target_hit": false,
|
||||
"line_precision": 0.0,
|
||||
"line_recall": 0.0,
|
||||
"line_f1": 0.0,
|
||||
"returned_lines": 31,
|
||||
"overlap_lines": 0
|
||||
},
|
||||
"status": "completed",
|
||||
"elapsed_seconds": 18.156966786002158,
|
||||
"rounds": 3,
|
||||
"tool_calls": 3,
|
||||
"tool_errors": 0,
|
||||
"invalid_actions": 0,
|
||||
"input_tokens": 7705,
|
||||
"output_tokens": 89,
|
||||
"returned_chars": 1390,
|
||||
"references": [
|
||||
{
|
||||
"path": "src/flask/helpers.py",
|
||||
"start_line": 318,
|
||||
"end_line": 348,
|
||||
"sha256": "ac96607bbfc9dbcf09d5442fe7e90d7f8a046b0df7d8f3fc3535b44112016d71",
|
||||
"verified": true
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "flask-09",
|
||||
"repository": "flask",
|
||||
"scores": {
|
||||
"file_hit": false,
|
||||
"target_hit": false,
|
||||
"line_precision": 0.0,
|
||||
"line_recall": 0.0,
|
||||
"line_f1": 0.0,
|
||||
"returned_lines": 21,
|
||||
"overlap_lines": 0
|
||||
},
|
||||
"status": "completed",
|
||||
"elapsed_seconds": 16.69060758499836,
|
||||
"rounds": 3,
|
||||
"tool_calls": 3,
|
||||
"tool_errors": 0,
|
||||
"invalid_actions": 0,
|
||||
"input_tokens": 7582,
|
||||
"output_tokens": 90,
|
||||
"returned_chars": 747,
|
||||
"references": [
|
||||
{
|
||||
"path": "src/flask/app.py",
|
||||
"start_line": 42,
|
||||
"end_line": 62,
|
||||
"sha256": "5c6aa0151b0b8018732280761d27eda0c4c83378630e21faadd57b73783720a7",
|
||||
"verified": true
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "flask-10",
|
||||
"repository": "flask",
|
||||
"scores": {
|
||||
"file_hit": false,
|
||||
"target_hit": false,
|
||||
"line_precision": 0.0,
|
||||
"line_recall": 0.0,
|
||||
"line_f1": 0.0,
|
||||
"returned_lines": 0,
|
||||
"overlap_lines": 0
|
||||
},
|
||||
"status": "budget_exhausted",
|
||||
"elapsed_seconds": 38.21693752400461,
|
||||
"rounds": 6,
|
||||
"tool_calls": 6,
|
||||
"tool_errors": 2,
|
||||
"invalid_actions": 1,
|
||||
"input_tokens": 15330,
|
||||
"output_tokens": 170,
|
||||
"returned_chars": 0,
|
||||
"references": []
|
||||
},
|
||||
{
|
||||
"id": "click-01",
|
||||
"repository": "click",
|
||||
"scores": {
|
||||
"file_hit": false,
|
||||
"target_hit": false,
|
||||
"line_precision": 0.0,
|
||||
"line_recall": 0.0,
|
||||
"line_f1": 0.0,
|
||||
"returned_lines": 111,
|
||||
"overlap_lines": 0
|
||||
},
|
||||
"status": "completed",
|
||||
"elapsed_seconds": 24.070387410996773,
|
||||
"rounds": 3,
|
||||
"tool_calls": 3,
|
||||
"tool_errors": 0,
|
||||
"invalid_actions": 0,
|
||||
"input_tokens": 6917,
|
||||
"output_tokens": 99,
|
||||
"returned_chars": 3619,
|
||||
"references": [
|
||||
{
|
||||
"path": "src/click/_termui_impl.py",
|
||||
"start_line": 430,
|
||||
"end_line": 540,
|
||||
"sha256": "0125e12e2f4840873426cf4a4124bedfe48b65c3deb7756acac05ff5681b6e92",
|
||||
"verified": true
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "click-02",
|
||||
"repository": "click",
|
||||
"scores": {
|
||||
"file_hit": false,
|
||||
"target_hit": false,
|
||||
"line_precision": 0.0,
|
||||
"line_recall": 0.0,
|
||||
"line_f1": 0.0,
|
||||
"returned_lines": 89,
|
||||
"overlap_lines": 0
|
||||
},
|
||||
"status": "completed",
|
||||
"elapsed_seconds": 19.466111501998967,
|
||||
"rounds": 3,
|
||||
"tool_calls": 3,
|
||||
"tool_errors": 0,
|
||||
"invalid_actions": 0,
|
||||
"input_tokens": 8056,
|
||||
"output_tokens": 90,
|
||||
"returned_chars": 2767,
|
||||
"references": [
|
||||
{
|
||||
"path": "tests/test_basic.py",
|
||||
"start_line": 271,
|
||||
"end_line": 359,
|
||||
"sha256": "44bbbbf8eb708cf11ead3717d22a2bf910dd65624a4b35d9f326e1a937aee329",
|
||||
"verified": true
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "click-03",
|
||||
"repository": "click",
|
||||
"scores": {
|
||||
"file_hit": false,
|
||||
"target_hit": false,
|
||||
"line_precision": 0.0,
|
||||
"line_recall": 0.0,
|
||||
"line_f1": 0.0,
|
||||
"returned_lines": 21,
|
||||
"overlap_lines": 0
|
||||
},
|
||||
"status": "completed",
|
||||
"elapsed_seconds": 18.362840486995992,
|
||||
"rounds": 3,
|
||||
"tool_calls": 3,
|
||||
"tool_errors": 0,
|
||||
"invalid_actions": 0,
|
||||
"input_tokens": 7364,
|
||||
"output_tokens": 91,
|
||||
"returned_chars": 974,
|
||||
"references": [
|
||||
{
|
||||
"path": "src/click/parser.py",
|
||||
"start_line": 282,
|
||||
"end_line": 302,
|
||||
"sha256": "9d4d40876a75d6adbdba5d6f35d53db303e8fcf52e2abc54e985d3a22ef5ab57",
|
||||
"verified": true
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "click-04",
|
||||
"repository": "click",
|
||||
"scores": {
|
||||
"file_hit": false,
|
||||
"target_hit": false,
|
||||
"line_precision": 0.0,
|
||||
"line_recall": 0.0,
|
||||
"line_f1": 0.0,
|
||||
"returned_lines": 3,
|
||||
"overlap_lines": 0
|
||||
},
|
||||
"status": "completed",
|
||||
"elapsed_seconds": 43.20344570299494,
|
||||
"rounds": 6,
|
||||
"tool_calls": 5,
|
||||
"tool_errors": 1,
|
||||
"invalid_actions": 1,
|
||||
"input_tokens": 18336,
|
||||
"output_tokens": 184,
|
||||
"returned_chars": 199,
|
||||
"references": [
|
||||
{
|
||||
"path": "src/click/parser.py",
|
||||
"start_line": 55,
|
||||
"end_line": 57,
|
||||
"sha256": "9d4d40876a75d6adbdba5d6f35d53db303e8fcf52e2abc54e985d3a22ef5ab57",
|
||||
"verified": true
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "click-05",
|
||||
"repository": "click",
|
||||
"scores": {
|
||||
"file_hit": false,
|
||||
"target_hit": false,
|
||||
"line_precision": 0.0,
|
||||
"line_recall": 0.0,
|
||||
"line_f1": 0.0,
|
||||
"returned_lines": 75,
|
||||
"overlap_lines": 0
|
||||
},
|
||||
"status": "completed",
|
||||
"elapsed_seconds": 24.201707656000508,
|
||||
"rounds": 3,
|
||||
"tool_calls": 3,
|
||||
"tool_errors": 0,
|
||||
"invalid_actions": 0,
|
||||
"input_tokens": 8051,
|
||||
"output_tokens": 93,
|
||||
"returned_chars": 3025,
|
||||
"references": [
|
||||
{
|
||||
"path": "src/click/__init__.py",
|
||||
"start_line": 3,
|
||||
"end_line": 77,
|
||||
"sha256": "e98c92d5a7b29276742d8c1e5a8ccd672d00f6767fd75c2660886fddc6d0ad8a",
|
||||
"verified": true
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "click-06",
|
||||
"repository": "click",
|
||||
"scores": {
|
||||
"file_hit": true,
|
||||
"target_hit": false,
|
||||
"line_precision": 0.0,
|
||||
"line_recall": 0.0,
|
||||
"line_f1": 0.0,
|
||||
"returned_lines": 87,
|
||||
"overlap_lines": 0
|
||||
},
|
||||
"status": "completed",
|
||||
"elapsed_seconds": 19.431287539999175,
|
||||
"rounds": 3,
|
||||
"tool_calls": 3,
|
||||
"tool_errors": 0,
|
||||
"invalid_actions": 0,
|
||||
"input_tokens": 7884,
|
||||
"output_tokens": 89,
|
||||
"returned_chars": 2667,
|
||||
"references": [
|
||||
{
|
||||
"path": "src/click/core.py",
|
||||
"start_line": 10,
|
||||
"end_line": 96,
|
||||
"sha256": "814869352f5c14119d1175dd8ac1951be791bc67f8f514f215a831ba5ab0745c",
|
||||
"verified": true
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "click-07",
|
||||
"repository": "click",
|
||||
"scores": {
|
||||
"file_hit": false,
|
||||
"target_hit": false,
|
||||
"line_precision": 0.0,
|
||||
"line_recall": 0.0,
|
||||
"line_f1": 0.0,
|
||||
"returned_lines": 56,
|
||||
"overlap_lines": 0
|
||||
},
|
||||
"status": "completed",
|
||||
"elapsed_seconds": 41.42516288800107,
|
||||
"rounds": 6,
|
||||
"tool_calls": 5,
|
||||
"tool_errors": 2,
|
||||
"invalid_actions": 1,
|
||||
"input_tokens": 17088,
|
||||
"output_tokens": 198,
|
||||
"returned_chars": 1747,
|
||||
"references": [
|
||||
{
|
||||
"path": "src/click/parser.py",
|
||||
"start_line": 54,
|
||||
"end_line": 109,
|
||||
"sha256": "9d4d40876a75d6adbdba5d6f35d53db303e8fcf52e2abc54e985d3a22ef5ab57",
|
||||
"verified": true
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "click-08",
|
||||
"repository": "click",
|
||||
"scores": {
|
||||
"file_hit": false,
|
||||
"target_hit": false,
|
||||
"line_precision": 0.0,
|
||||
"line_recall": 0.0,
|
||||
"line_f1": 0.0,
|
||||
"returned_lines": 14,
|
||||
"overlap_lines": 0
|
||||
},
|
||||
"status": "completed",
|
||||
"elapsed_seconds": 18.4918164289993,
|
||||
"rounds": 3,
|
||||
"tool_calls": 3,
|
||||
"tool_errors": 0,
|
||||
"invalid_actions": 0,
|
||||
"input_tokens": 6553,
|
||||
"output_tokens": 93,
|
||||
"returned_chars": 487,
|
||||
"references": [
|
||||
{
|
||||
"path": "src/click/decorators.py",
|
||||
"start_line": 381,
|
||||
"end_line": 394,
|
||||
"sha256": "e4feda6e126d010629fca1e08d4be132fe3ae044703b3aefd9e9cd9279701f24",
|
||||
"verified": true
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "click-09",
|
||||
"repository": "click",
|
||||
"scores": {
|
||||
"file_hit": false,
|
||||
"target_hit": false,
|
||||
"line_precision": 0.0,
|
||||
"line_recall": 0.0,
|
||||
"line_f1": 0.0,
|
||||
"returned_lines": 21,
|
||||
"overlap_lines": 0
|
||||
},
|
||||
"status": "completed",
|
||||
"elapsed_seconds": 31.15418404900265,
|
||||
"rounds": 5,
|
||||
"tool_calls": 4,
|
||||
"tool_errors": 1,
|
||||
"invalid_actions": 1,
|
||||
"input_tokens": 11614,
|
||||
"output_tokens": 163,
|
||||
"returned_chars": 1027,
|
||||
"references": [
|
||||
{
|
||||
"path": "src/click/formatting.py",
|
||||
"start_line": 38,
|
||||
"end_line": 58,
|
||||
"sha256": "061ab1e105dd290f56e162a49c8c23e4a3ca166b5db863ae1aad72c3f4c72d9f",
|
||||
"verified": true
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "click-10",
|
||||
"repository": "click",
|
||||
"scores": {
|
||||
"file_hit": false,
|
||||
"target_hit": false,
|
||||
"line_precision": 0.0,
|
||||
"line_recall": 0.0,
|
||||
"line_f1": 0.0,
|
||||
"returned_lines": 48,
|
||||
"overlap_lines": 0
|
||||
},
|
||||
"status": "completed",
|
||||
"elapsed_seconds": 18.543147828000656,
|
||||
"rounds": 3,
|
||||
"tool_calls": 3,
|
||||
"tool_errors": 0,
|
||||
"invalid_actions": 0,
|
||||
"input_tokens": 7507,
|
||||
"output_tokens": 93,
|
||||
"returned_chars": 1353,
|
||||
"references": [
|
||||
{
|
||||
"path": "src/click/_compat.py",
|
||||
"start_line": 22,
|
||||
"end_line": 69,
|
||||
"sha256": "bf7c4166415bbc0d415cf46415f04973afa92303c8ef7e399be910127a5502ce",
|
||||
"verified": true
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
@@ -0,0 +1,490 @@
|
||||
{
|
||||
"suite": {
|
||||
"name": "live-search-v1",
|
||||
"scope": "30 hand-authored English single-function localization tasks in three public Python repositories. Development benchmark, not a downstream coding-task or contamination-free evaluation.",
|
||||
"repositories": {
|
||||
"requests": {
|
||||
"commit": "b25c87d7cb8d6a18a37fa12442b5f883f9e41741",
|
||||
"url": "https://github.com/psf/requests.git"
|
||||
},
|
||||
"flask": {
|
||||
"commit": "2c1b30d0503cfb064f1cb252e6614a06915a362a",
|
||||
"url": "https://github.com/pallets/flask.git"
|
||||
},
|
||||
"click": {
|
||||
"commit": "fd183b2ced1cb5857784fe7fb22f4982f671f098",
|
||||
"url": "https://github.com/pallets/click.git"
|
||||
}
|
||||
},
|
||||
"tasks": [
|
||||
{
|
||||
"id": "requests-01",
|
||||
"repository": "requests",
|
||||
"query": "Find where an unsuccessful HTTP status becomes an exception containing the server reason and URL.",
|
||||
"targets": [
|
||||
{
|
||||
"path": "src/requests/models.py",
|
||||
"symbol": "Response.raise_for_status",
|
||||
"start_line": 1002,
|
||||
"end_line": 1026,
|
||||
"sha256": "32365d67893bb67c3ed67cf93ca4a18e63e6ab29342fa0dc8b09c59e06ff564e"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "requests-02",
|
||||
"repository": "requests",
|
||||
"query": "Locate the response iterator that preserves an incomplete trailing line across downloaded chunks.",
|
||||
"targets": [
|
||||
{
|
||||
"path": "src/requests/models.py",
|
||||
"symbol": "Response.iter_lines",
|
||||
"start_line": 867,
|
||||
"end_line": 888,
|
||||
"sha256": "32365d67893bb67c3ed67cf93ca4a18e63e6ab29342fa0dc8b09c59e06ff564e"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "requests-03",
|
||||
"repository": "requests",
|
||||
"query": "Where is it decided whether credentials may survive a redirect to another hostname, port, or protocol?",
|
||||
"targets": [
|
||||
{
|
||||
"path": "src/requests/sessions.py",
|
||||
"symbol": "SessionRedirectMixin.should_strip_auth",
|
||||
"start_line": 129,
|
||||
"end_line": 157,
|
||||
"sha256": "0a5d5da449ce7f0af3ccf6e4bbe7a67a935e37846dff4ff9f08cb6c7e2464e6f"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "requests-04",
|
||||
"repository": "requests",
|
||||
"query": "Find the logic that changes the HTTP verb when following 301, 302, or 303 redirects.",
|
||||
"targets": [
|
||||
{
|
||||
"path": "src/requests/sessions.py",
|
||||
"symbol": "SessionRedirectMixin.rebuild_method",
|
||||
"start_line": 337,
|
||||
"end_line": 353,
|
||||
"sha256": "0a5d5da449ce7f0af3ccf6e4bbe7a67a935e37846dff4ff9f08cb6c7e2464e6f"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "requests-05",
|
||||
"repository": "requests",
|
||||
"query": "Locate where environment proxy settings and certificate bundle variables are merged with session options.",
|
||||
"targets": [
|
||||
{
|
||||
"path": "src/requests/sessions.py",
|
||||
"symbol": "Session.merge_environment_settings",
|
||||
"start_line": 757,
|
||||
"end_line": 779,
|
||||
"sha256": "0a5d5da449ce7f0af3ccf6e4bbe7a67a935e37846dff4ff9f08cb6c7e2464e6f"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "requests-06",
|
||||
"repository": "requests",
|
||||
"query": "Find where a 401 digest challenge causes the original request to be copied and sent again with authentication.",
|
||||
"targets": [
|
||||
{
|
||||
"path": "src/requests/auth.py",
|
||||
"symbol": "HTTPDigestAuth.handle_401",
|
||||
"start_line": 250,
|
||||
"end_line": 283,
|
||||
"sha256": "905ef9b6a9cb72d67d31ffe19bd4d9223e1c4169cde6ec51cfca16b31e70991d"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "requests-07",
|
||||
"repository": "requests",
|
||||
"query": "Locate the helper that fills a cookie container from a mapping while optionally preserving existing names.",
|
||||
"targets": [
|
||||
{
|
||||
"path": "src/requests/cookies.py",
|
||||
"symbol": "cookiejar_from_dict",
|
||||
"start_line": 530,
|
||||
"end_line": 539,
|
||||
"sha256": "6cd8be8aa123e0d3d9d34fa86feac7bf392f39bccdde5129830de0ea9692dd7c"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "requests-08",
|
||||
"repository": "requests",
|
||||
"query": "Find where credentials are loaded from the users netrc file, including NETRC and home-directory lookup.",
|
||||
"targets": [
|
||||
{
|
||||
"path": "src/requests/utils.py",
|
||||
"symbol": "get_netrc_auth",
|
||||
"start_line": 210,
|
||||
"end_line": 248,
|
||||
"sha256": "5aa53ceab677c2f842fad42359c8ed1ff1c4299c1607789609957a496e4311d4"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "requests-09",
|
||||
"repository": "requests",
|
||||
"query": "Find the implementation that seeks a request body back to its saved position and fails for an unrewindable stream.",
|
||||
"targets": [
|
||||
{
|
||||
"path": "src/requests/utils.py",
|
||||
"symbol": "rewind_body",
|
||||
"start_line": 1075,
|
||||
"end_line": 1086,
|
||||
"sha256": "5aa53ceab677c2f842fad42359c8ed1ff1c4299c1607789609957a496e4311d4"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "requests-10",
|
||||
"repository": "requests",
|
||||
"query": "Where is a request URL validated, its international hostname encoded, and its query parameters appended?",
|
||||
"targets": [
|
||||
{
|
||||
"path": "src/requests/models.py",
|
||||
"symbol": "PreparedRequest.prepare_url",
|
||||
"start_line": 416,
|
||||
"end_line": 481,
|
||||
"sha256": "32365d67893bb67c3ed67cf93ca4a18e63e6ab29342fa0dc8b09c59e06ff564e"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "flask-01",
|
||||
"repository": "flask",
|
||||
"query": "Find the code that invokes the view function selected by the matched URL rule.",
|
||||
"targets": [
|
||||
{
|
||||
"path": "src/flask/app.py",
|
||||
"symbol": "Flask.dispatch_request",
|
||||
"start_line": 889,
|
||||
"end_line": 902,
|
||||
"sha256": "5c6aa0151b0b8018732280761d27eda0c4c83378630e21faadd57b73783720a7"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "flask-02",
|
||||
"repository": "flask",
|
||||
"query": "Where are view return values such as tuples, dictionaries and strings converted into a response object?",
|
||||
"targets": [
|
||||
{
|
||||
"path": "src/flask/app.py",
|
||||
"symbol": "Flask.make_response",
|
||||
"start_line": 1186,
|
||||
"end_line": 1269,
|
||||
"sha256": "5c6aa0151b0b8018732280761d27eda0c4c83378630e21faadd57b73783720a7"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "flask-03",
|
||||
"repository": "flask",
|
||||
"query": "Find where coroutine view functions are adapted for synchronous request handling.",
|
||||
"targets": [
|
||||
{
|
||||
"path": "src/flask/app.py",
|
||||
"symbol": "Flask.ensure_sync",
|
||||
"start_line": 975,
|
||||
"end_line": 978,
|
||||
"sha256": "5c6aa0151b0b8018732280761d27eda0c4c83378630e21faadd57b73783720a7"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "flask-04",
|
||||
"repository": "flask",
|
||||
"query": "Locate the processing that runs after-request callbacks and saves the session before returning the response.",
|
||||
"targets": [
|
||||
{
|
||||
"path": "src/flask/app.py",
|
||||
"symbol": "Flask.process_response",
|
||||
"start_line": 1311,
|
||||
"end_line": 1324,
|
||||
"sha256": "5c6aa0151b0b8018732280761d27eda0c4c83378630e21faadd57b73783720a7"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "flask-05",
|
||||
"repository": "flask",
|
||||
"query": "Find where before-request handlers can short-circuit normal request dispatch by returning a value.",
|
||||
"targets": [
|
||||
{
|
||||
"path": "src/flask/app.py",
|
||||
"symbol": "Flask.preprocess_request",
|
||||
"start_line": 1281,
|
||||
"end_line": 1296,
|
||||
"sha256": "5c6aa0151b0b8018732280761d27eda0c4c83378630e21faadd57b73783720a7"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "flask-06",
|
||||
"repository": "flask",
|
||||
"query": "Where is a streaming generator wrapped so the request context remains active while producing its items?",
|
||||
"targets": [
|
||||
{
|
||||
"path": "src/flask/helpers.py",
|
||||
"symbol": "stream_with_context",
|
||||
"start_line": 104,
|
||||
"end_line": 143,
|
||||
"sha256": "ac96607bbfc9dbcf09d5442fe7e90d7f8a046b0df7d8f3fc3535b44112016d71"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "flask-07",
|
||||
"repository": "flask",
|
||||
"query": "Find where a categorized one-time message is stored in the session and a notification signal is emitted.",
|
||||
"targets": [
|
||||
{
|
||||
"path": "src/flask/helpers.py",
|
||||
"symbol": "flash",
|
||||
"start_line": 340,
|
||||
"end_line": 349,
|
||||
"sha256": "ac96607bbfc9dbcf09d5442fe7e90d7f8a046b0df7d8f3fc3535b44112016d71"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "flask-08",
|
||||
"repository": "flask",
|
||||
"query": "Locate where stored one-time messages are popped from the session, cached for the request, and filtered by category.",
|
||||
"targets": [
|
||||
{
|
||||
"path": "src/flask/helpers.py",
|
||||
"symbol": "get_flashed_messages",
|
||||
"start_line": 383,
|
||||
"end_line": 391,
|
||||
"sha256": "ac96607bbfc9dbcf09d5442fe7e90d7f8a046b0df7d8f3fc3535b44112016d71"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "flask-09",
|
||||
"repository": "flask",
|
||||
"query": "Find where the command-line startup reads dotenv files without replacing environment variables already set.",
|
||||
"targets": [
|
||||
{
|
||||
"path": "src/flask/cli.py",
|
||||
"symbol": "load_dotenv",
|
||||
"start_line": 740,
|
||||
"end_line": 771,
|
||||
"sha256": "3df87bdbe07196fa07d101c20ce7351ef5c6ecaab95deb5fdf3ccdcf690d4879"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "flask-10",
|
||||
"repository": "flask",
|
||||
"query": "Locate where a signed session cookie is verified and an invalid signature produces an empty session.",
|
||||
"targets": [
|
||||
{
|
||||
"path": "src/flask/sessions.py",
|
||||
"symbol": "SecureCookieSessionInterface.open_session",
|
||||
"start_line": 338,
|
||||
"end_line": 349,
|
||||
"sha256": "76ebd81a608687f1f772032ecb6a1e4a3ac2b02bcbca517808128e5e9374426b"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "click-01",
|
||||
"repository": "click",
|
||||
"query": "Find where a supplied option value is normalized and matched against the allowed choices.",
|
||||
"targets": [
|
||||
{
|
||||
"path": "src/click/types.py",
|
||||
"symbol": "Choice.convert",
|
||||
"start_line": 344,
|
||||
"end_line": 358,
|
||||
"sha256": "2814d1c4ddbc711d5567999bf62257f7c3104b0fe9f521b3963a9f108f33e53c"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "click-02",
|
||||
"repository": "click",
|
||||
"query": "Locate where a filesystem argument is checked for existence, readability, writability, and allowed file or directory type.",
|
||||
"targets": [
|
||||
{
|
||||
"path": "src/click/types.py",
|
||||
"symbol": "Path.convert",
|
||||
"start_line": 930,
|
||||
"end_line": 995,
|
||||
"sha256": "2814d1c4ddbc711d5567999bf62257f7c3104b0fe9f521b3963a9f108f33e53c"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "click-03",
|
||||
"repository": "click",
|
||||
"query": "Find where a date argument is parsed by trying several accepted formats and reports a failure if none match.",
|
||||
"targets": [
|
||||
{
|
||||
"path": "src/click/types.py",
|
||||
"symbol": "DateTime.convert",
|
||||
"start_line": 448,
|
||||
"end_line": 466,
|
||||
"sha256": "2814d1c4ddbc711d5567999bf62257f7c3104b0fe9f521b3963a9f108f33e53c"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "click-04",
|
||||
"repository": "click",
|
||||
"query": "Where is an option value from an environment variable split and grouped for multiple arguments?",
|
||||
"targets": [
|
||||
{
|
||||
"path": "src/click/core.py",
|
||||
"symbol": "Option.value_from_envvar",
|
||||
"start_line": 2983,
|
||||
"end_line": 2996,
|
||||
"sha256": "814869352f5c14119d1175dd8ac1951be791bc67f8f514f215a831ba5ab0745c"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "click-05",
|
||||
"repository": "click",
|
||||
"query": "Locate where a command group resolves a subcommand name, retries normalized names, and rejects unknown commands.",
|
||||
"targets": [
|
||||
{
|
||||
"path": "src/click/core.py",
|
||||
"symbol": "Group.resolve_command",
|
||||
"start_line": 1867,
|
||||
"end_line": 1889,
|
||||
"sha256": "814869352f5c14119d1175dd8ac1951be791bc67f8f514f215a831ba5ab0745c"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "click-06",
|
||||
"repository": "click",
|
||||
"query": "Find the helper that enters a context manager and registers its cleanup with the command context.",
|
||||
"targets": [
|
||||
{
|
||||
"path": "src/click/core.py",
|
||||
"symbol": "Context.with_resource",
|
||||
"start_line": 598,
|
||||
"end_line": 598,
|
||||
"sha256": "814869352f5c14119d1175dd8ac1951be791bc67f8f514f215a831ba5ab0745c"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "click-07",
|
||||
"repository": "click",
|
||||
"query": "Find where each element of a tuple argument is converted using its corresponding parameter type.",
|
||||
"targets": [
|
||||
{
|
||||
"path": "src/click/types.py",
|
||||
"symbol": "Tuple.convert",
|
||||
"start_line": 1049,
|
||||
"end_line": 1065,
|
||||
"sha256": "2814d1c4ddbc711d5567999bf62257f7c3104b0fe9f521b3963a9f108f33e53c"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "click-08",
|
||||
"repository": "click",
|
||||
"query": "Locate the interactive yes-or-no question loop that retries invalid answers and can abort on rejection.",
|
||||
"targets": [
|
||||
{
|
||||
"path": "src/click/termui.py",
|
||||
"symbol": "confirm",
|
||||
"start_line": 223,
|
||||
"end_line": 252,
|
||||
"sha256": "bc062b282d9aedffcd7c42210138445587dafff89be5741b4d2086b499400510"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "click-09",
|
||||
"repository": "click",
|
||||
"query": "Find the testing helper that temporarily changes the current directory and removes its temporary directory on exit.",
|
||||
"targets": [
|
||||
{
|
||||
"path": "src/click/testing.py",
|
||||
"symbol": "CliRunner.isolated_filesystem",
|
||||
"start_line": 552,
|
||||
"end_line": 565,
|
||||
"sha256": "d9e2dd01a0890864e83f94f0f9737c263ac36f7c1b2487c047183777472b5b93"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "click-10",
|
||||
"repository": "click",
|
||||
"query": "Where does a lazily opened file acquire its actual stream and translate operating-system errors into a file error?",
|
||||
"targets": [
|
||||
{
|
||||
"path": "src/click/utils.py",
|
||||
"symbol": "LazyFile.open",
|
||||
"start_line": 156,
|
||||
"end_line": 167,
|
||||
"sha256": "6f5326faeb040c11ed13070f96d3c89f7cd22b89f0d8a4e9e460bbfe849394ba"
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
},
|
||||
"suite_sha256": "d06effdf41ee2e38bf8f44629949d9b4491067986fc5a73aef22cf80c5a5bf96",
|
||||
"model": {
|
||||
"model": "openbmb/MiniCPM5-1B+best",
|
||||
"base_id": "openbmb/MiniCPM5-1B",
|
||||
"base_revision": "87179e5c1f455ef22e6223592d2d61351b525bfc",
|
||||
"backend": "transformers",
|
||||
"quantization": "nf4",
|
||||
"adapter": "runs/minicpm5-policy-v2/best",
|
||||
"load_seconds": 9.676906742999563
|
||||
},
|
||||
"source_sha256": {
|
||||
"__init__.py": "dcd2b573883b8068e806e3052adf9c03728f1cf621d35ff013179de76f76c702",
|
||||
"text.py": "8db7b80ee446480175f7691872303c8b72fd014604ad4fb575b359cee71f1938",
|
||||
"io.py": "b29c21db767ea9773281b17fe0754375153d18fb64f314907528f00d584a047f",
|
||||
"data.py": "20a891eff11d555d2301b62f4953146603efc87b03da4802f136f694ebdba78d",
|
||||
"encoder.py": "cb5671c08f28a70ffd0e2a7033528efc3581896a27242d55eb250ce922ab038f",
|
||||
"metrics.py": "cb93d30878c0c1006f06d1d0c4cbb188f90c6f4fc65f69f9a3b11d847f28ad27",
|
||||
"train.py": "cf20f6189967cfc88f3849bc3a225f2680c42026d53f204d9d5afc05fe88a2d6",
|
||||
"lexical.py": "6c47c834c6bbf7b317e2514057eee7444720581f5ec452ecb01ac666397b86b3",
|
||||
"symbols.py": "a377d6aa2c6b479c2ee115dd8e4859f0c06d188ae97b86fa1c7133885ee32d86",
|
||||
"index.py": "e8b0a666b41fed48d98616d5dcdd104e59c4308ac990281b463e5ee68f97877f",
|
||||
"evaluate.py": "29c942b9cbfff21ba8514511e27752527f0ec5e2cad5fbf57ec3b220fc7acd0b",
|
||||
"__main__.py": "ab30d9b696e41ece67094e19c87b1a31fc0ebce8d3afa79dc8eaf561f8fb4895",
|
||||
"download.py": "24f9802fb7cc82b81f6b1cb93e9244352d7b86feeb0a54ec055fb3e8d0a6a4d7",
|
||||
"cli.py": "d081962fc459808c65bd4c1d389e12ad9a9466df0a1374c8095a8bd5ccbb369e",
|
||||
"scout.py": "c3150baae51f37646305fee1d2c61e7b37f2ec9c4a0ba7f92ab4d7d6b1710161",
|
||||
"server.py": "483fe8051012249e38a6836cbe217d8c581d033e4d19026a29a7bd5c83878190",
|
||||
"live_tools.py": "e10fb6aaeaa6d7c4a5a35c0c6a210be4a4c283b18d215767d6bee0bac707d14c",
|
||||
"local_policy.py": "b0711b46a6399dfbeeecdcc045fb7392030e187d7e21c6e4fa9b51e6fc8056f4",
|
||||
"agent.py": "47357a900e9696b7ec5268eb031067eb0069faaeef92fd3159bd822d6bc67cdb",
|
||||
"live_server.py": "ccfb3c63115fe991c2f5f22c05f794d89c90d783b52f8ddbb668a8541e3af6fe",
|
||||
"native_protocol.py": "3a4d16c1d2cd90cb7d81e635f40d721270b0d541f454557a6d14e80b6b1073a7",
|
||||
"eval_live.py": "38250d45fd7e60c23c4c86f5819242736f29df8af384f92713128c84ffd734ff",
|
||||
"prepare_live.py": "0675173c5e9c116c995c271dc06de914388f19eb83779c07e3009000a810290b",
|
||||
"live_data.py": "4d064db6c4da6ef3b55c63e92c0da4ceb62d5cecd746458a90e0b7a9b4e86913",
|
||||
"transformers_policy.py": "e8e8a81b70eb37563973e640534b425a7ea9fc7d398a39dc8d8f71b9be3f1b89",
|
||||
"train_policy.py": "d712111e3b7272372e68986f62c464afcb66d0c4b13b2cdd632459980e2091a8",
|
||||
"keyword_baseline.py": "5fe18c76b4c6e085c358ab5c490d717ffdd82b917a2882cf826fa26c57f64816"
|
||||
},
|
||||
"max_rounds": 6,
|
||||
"max_chars": 6000,
|
||||
"timeout_seconds": 90,
|
||||
"context": 8192,
|
||||
"max_generation_tokens": 512,
|
||||
"tokenizer": "pinned_hf",
|
||||
"temperature": 0,
|
||||
"seed": 42,
|
||||
"scope": "Single-function localization; no large-model baseline; public repositories may have appeared in the base model's pretraining."
|
||||
}
|
||||
@@ -0,0 +1,15 @@
|
||||
{
|
||||
"selected_step": 104,
|
||||
"base_id": "openbmb/MiniCPM5-1B",
|
||||
"base_revision": "87179e5c1f455ef22e6223592d2d61351b525bfc",
|
||||
"files": {
|
||||
"best/adapter_config.json": {
|
||||
"sha256": "e2234e69e62ebdc618a7e6cc0881a962e31f86a859b8b9b0f5c83823a317c15a",
|
||||
"bytes": 969
|
||||
},
|
||||
"best/adapter_model.safetensors": {
|
||||
"sha256": "1ef1ef35fc5545d3d8ed64d5be182e43759d96704e6a07afba6bde291bcb89bc",
|
||||
"bytes": 44871152
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,89 @@
|
||||
{
|
||||
"tasks": 30,
|
||||
"target_hit_rate": 0,
|
||||
"file_hit_rate": 0.1,
|
||||
"macro_line_precision": 0.0,
|
||||
"macro_line_recall": 0.0,
|
||||
"macro_line_f1": 0.0,
|
||||
"latency_median_seconds": 32.87817524649654,
|
||||
"latency_p95_seconds": 44.770849431097304,
|
||||
"statuses": {
|
||||
"budget_exhausted": 23,
|
||||
"completed": 7
|
||||
},
|
||||
"total_tool_errors": 22,
|
||||
"total_invalid_actions": 51,
|
||||
"mean_rounds": 5.3,
|
||||
"mean_tool_calls": 4.366666666666666,
|
||||
"total_input_tokens": 464686,
|
||||
"total_output_tokens": 4643,
|
||||
"mean_returned_chars": 419.6,
|
||||
"per_repository": {
|
||||
"requests": {
|
||||
"tasks": 10,
|
||||
"target_hit_rate": 0,
|
||||
"file_hit_rate": 0.2,
|
||||
"macro_line_precision": 0.0,
|
||||
"macro_line_recall": 0.0,
|
||||
"macro_line_f1": 0.0,
|
||||
"latency_median_seconds": 30.96327949500119,
|
||||
"latency_p95_seconds": 35.69990707220022,
|
||||
"statuses": {
|
||||
"budget_exhausted": 6,
|
||||
"completed": 4
|
||||
},
|
||||
"total_tool_errors": 6,
|
||||
"total_invalid_actions": 14,
|
||||
"mean_rounds": 4.8,
|
||||
"mean_tool_calls": 4,
|
||||
"total_input_tokens": 135648,
|
||||
"total_output_tokens": 1407,
|
||||
"mean_returned_chars": 774.7
|
||||
},
|
||||
"flask": {
|
||||
"tasks": 10,
|
||||
"target_hit_rate": 0,
|
||||
"file_hit_rate": 0.1,
|
||||
"macro_line_precision": 0.0,
|
||||
"macro_line_recall": 0.0,
|
||||
"macro_line_f1": 0.0,
|
||||
"latency_median_seconds": 31.488584604001517,
|
||||
"latency_p95_seconds": 34.95061498904797,
|
||||
"statuses": {
|
||||
"budget_exhausted": 8,
|
||||
"completed": 2
|
||||
},
|
||||
"total_tool_errors": 7,
|
||||
"total_invalid_actions": 18,
|
||||
"mean_rounds": 5.4,
|
||||
"mean_tool_calls": 4.4,
|
||||
"total_input_tokens": 161045,
|
||||
"total_output_tokens": 1584,
|
||||
"mean_returned_chars": 238.9
|
||||
},
|
||||
"click": {
|
||||
"tasks": 10,
|
||||
"target_hit_rate": 0,
|
||||
"file_hit_rate": 0,
|
||||
"macro_line_precision": 0.0,
|
||||
"macro_line_recall": 0.0,
|
||||
"macro_line_f1": 0.0,
|
||||
"latency_median_seconds": 43.22066144999917,
|
||||
"latency_p95_seconds": 46.640672169248504,
|
||||
"statuses": {
|
||||
"budget_exhausted": 9,
|
||||
"completed": 1
|
||||
},
|
||||
"total_tool_errors": 9,
|
||||
"total_invalid_actions": 19,
|
||||
"mean_rounds": 5.7,
|
||||
"mean_tool_calls": 4.7,
|
||||
"total_input_tokens": 167993,
|
||||
"total_output_tokens": 1652,
|
||||
"mean_returned_chars": 245.2
|
||||
}
|
||||
},
|
||||
"sampled_peak_gpu_memory_mib": null,
|
||||
"gpu_sampling_interval_seconds": 1,
|
||||
"execution_note": "The process was interrupted after 28 persisted tasks. Only the remaining two tasks were run after verifying the frozen suite, source, repository, and adapter hashes, with a fresh model warm-up. Original in-memory GPU samples were lost; no full-run GPU peak is reported."
|
||||
}
|
||||
@@ -0,0 +1,748 @@
|
||||
[
|
||||
{
|
||||
"id": "requests-01",
|
||||
"repository": "requests",
|
||||
"scores": {
|
||||
"file_hit": false,
|
||||
"target_hit": false,
|
||||
"line_precision": 0.0,
|
||||
"line_recall": 0.0,
|
||||
"line_f1": 0.0,
|
||||
"returned_lines": 0,
|
||||
"overlap_lines": 0
|
||||
},
|
||||
"status": "budget_exhausted",
|
||||
"elapsed_seconds": 32.45663743799378,
|
||||
"rounds": 6,
|
||||
"tool_calls": 5,
|
||||
"tool_errors": 1,
|
||||
"invalid_actions": 2,
|
||||
"input_tokens": 17782,
|
||||
"output_tokens": 171,
|
||||
"returned_chars": 0,
|
||||
"references": []
|
||||
},
|
||||
{
|
||||
"id": "requests-02",
|
||||
"repository": "requests",
|
||||
"scores": {
|
||||
"file_hit": false,
|
||||
"target_hit": false,
|
||||
"line_precision": 0.0,
|
||||
"line_recall": 0.0,
|
||||
"line_f1": 0.0,
|
||||
"returned_lines": 54,
|
||||
"overlap_lines": 0
|
||||
},
|
||||
"status": "completed",
|
||||
"elapsed_seconds": 15.850956338996184,
|
||||
"rounds": 3,
|
||||
"tool_calls": 3,
|
||||
"tool_errors": 0,
|
||||
"invalid_actions": 0,
|
||||
"input_tokens": 6817,
|
||||
"output_tokens": 89,
|
||||
"returned_chars": 1645,
|
||||
"references": [
|
||||
{
|
||||
"path": "src/requests/utils.py",
|
||||
"start_line": 206,
|
||||
"end_line": 259,
|
||||
"sha256": "5aa53ceab677c2f842fad42359c8ed1ff1c4299c1607789609957a496e4311d4",
|
||||
"verified": true
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "requests-03",
|
||||
"repository": "requests",
|
||||
"scores": {
|
||||
"file_hit": true,
|
||||
"target_hit": false,
|
||||
"line_precision": 0.0,
|
||||
"line_recall": 0.0,
|
||||
"line_f1": 0.0,
|
||||
"returned_lines": 63,
|
||||
"overlap_lines": 0
|
||||
},
|
||||
"status": "completed",
|
||||
"elapsed_seconds": 16.54656796400377,
|
||||
"rounds": 3,
|
||||
"tool_calls": 3,
|
||||
"tool_errors": 0,
|
||||
"invalid_actions": 0,
|
||||
"input_tokens": 7469,
|
||||
"output_tokens": 92,
|
||||
"returned_chars": 2636,
|
||||
"references": [
|
||||
{
|
||||
"path": "src/requests/sessions.py",
|
||||
"start_line": 282,
|
||||
"end_line": 344,
|
||||
"sha256": "0a5d5da449ce7f0af3ccf6e4bbe7a67a935e37846dff4ff9f08cb6c7e2464e6f",
|
||||
"verified": true
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "requests-04",
|
||||
"repository": "requests",
|
||||
"scores": {
|
||||
"file_hit": false,
|
||||
"target_hit": false,
|
||||
"line_precision": 0.0,
|
||||
"line_recall": 0.0,
|
||||
"line_f1": 0.0,
|
||||
"returned_lines": 0,
|
||||
"overlap_lines": 0
|
||||
},
|
||||
"status": "budget_exhausted",
|
||||
"elapsed_seconds": 30.534921878002933,
|
||||
"rounds": 6,
|
||||
"tool_calls": 5,
|
||||
"tool_errors": 1,
|
||||
"invalid_actions": 2,
|
||||
"input_tokens": 18341,
|
||||
"output_tokens": 169,
|
||||
"returned_chars": 0,
|
||||
"references": []
|
||||
},
|
||||
{
|
||||
"id": "requests-05",
|
||||
"repository": "requests",
|
||||
"scores": {
|
||||
"file_hit": false,
|
||||
"target_hit": false,
|
||||
"line_precision": 0.0,
|
||||
"line_recall": 0.0,
|
||||
"line_f1": 0.0,
|
||||
"returned_lines": 0,
|
||||
"overlap_lines": 0
|
||||
},
|
||||
"status": "budget_exhausted",
|
||||
"elapsed_seconds": 31.391637111999444,
|
||||
"rounds": 6,
|
||||
"tool_calls": 5,
|
||||
"tool_errors": 1,
|
||||
"invalid_actions": 2,
|
||||
"input_tokens": 18226,
|
||||
"output_tokens": 173,
|
||||
"returned_chars": 0,
|
||||
"references": []
|
||||
},
|
||||
{
|
||||
"id": "requests-06",
|
||||
"repository": "requests",
|
||||
"scores": {
|
||||
"file_hit": true,
|
||||
"target_hit": false,
|
||||
"line_precision": 0.0,
|
||||
"line_recall": 0.0,
|
||||
"line_f1": 0.0,
|
||||
"returned_lines": 19,
|
||||
"overlap_lines": 0
|
||||
},
|
||||
"status": "completed",
|
||||
"elapsed_seconds": 16.67756171700603,
|
||||
"rounds": 3,
|
||||
"tool_calls": 3,
|
||||
"tool_errors": 0,
|
||||
"invalid_actions": 0,
|
||||
"input_tokens": 7856,
|
||||
"output_tokens": 91,
|
||||
"returned_chars": 830,
|
||||
"references": [
|
||||
{
|
||||
"path": "src/requests/auth.py",
|
||||
"start_line": 285,
|
||||
"end_line": 303,
|
||||
"sha256": "905ef9b6a9cb72d67d31ffe19bd4d9223e1c4169cde6ec51cfca16b31e70991d",
|
||||
"verified": true
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "requests-07",
|
||||
"repository": "requests",
|
||||
"scores": {
|
||||
"file_hit": false,
|
||||
"target_hit": false,
|
||||
"line_precision": 0.0,
|
||||
"line_recall": 0.0,
|
||||
"line_f1": 0.0,
|
||||
"returned_lines": 0,
|
||||
"overlap_lines": 0
|
||||
},
|
||||
"status": "budget_exhausted",
|
||||
"elapsed_seconds": 34.047770837001735,
|
||||
"rounds": 6,
|
||||
"tool_calls": 5,
|
||||
"tool_errors": 1,
|
||||
"invalid_actions": 2,
|
||||
"input_tokens": 18003,
|
||||
"output_tokens": 171,
|
||||
"returned_chars": 0,
|
||||
"references": []
|
||||
},
|
||||
{
|
||||
"id": "requests-08",
|
||||
"repository": "requests",
|
||||
"scores": {
|
||||
"file_hit": false,
|
||||
"target_hit": false,
|
||||
"line_precision": 0.0,
|
||||
"line_recall": 0.0,
|
||||
"line_f1": 0.0,
|
||||
"returned_lines": 63,
|
||||
"overlap_lines": 0
|
||||
},
|
||||
"status": "completed",
|
||||
"elapsed_seconds": 16.234001347998856,
|
||||
"rounds": 3,
|
||||
"tool_calls": 3,
|
||||
"tool_errors": 0,
|
||||
"invalid_actions": 0,
|
||||
"input_tokens": 7472,
|
||||
"output_tokens": 92,
|
||||
"returned_chars": 2636,
|
||||
"references": [
|
||||
{
|
||||
"path": "src/requests/sessions.py",
|
||||
"start_line": 282,
|
||||
"end_line": 344,
|
||||
"sha256": "0a5d5da449ce7f0af3ccf6e4bbe7a67a935e37846dff4ff9f08cb6c7e2464e6f",
|
||||
"verified": true
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "requests-09",
|
||||
"repository": "requests",
|
||||
"scores": {
|
||||
"file_hit": false,
|
||||
"target_hit": false,
|
||||
"line_precision": 0.0,
|
||||
"line_recall": 0.0,
|
||||
"line_f1": 0.0,
|
||||
"returned_lines": 0,
|
||||
"overlap_lines": 0
|
||||
},
|
||||
"status": "budget_exhausted",
|
||||
"elapsed_seconds": 33.2997130549993,
|
||||
"rounds": 6,
|
||||
"tool_calls": 5,
|
||||
"tool_errors": 1,
|
||||
"invalid_actions": 2,
|
||||
"input_tokens": 17734,
|
||||
"output_tokens": 168,
|
||||
"returned_chars": 0,
|
||||
"references": []
|
||||
},
|
||||
{
|
||||
"id": "requests-10",
|
||||
"repository": "requests",
|
||||
"scores": {
|
||||
"file_hit": false,
|
||||
"target_hit": false,
|
||||
"line_precision": 0.0,
|
||||
"line_recall": 0.0,
|
||||
"line_f1": 0.0,
|
||||
"returned_lines": 0,
|
||||
"overlap_lines": 0
|
||||
},
|
||||
"status": "budget_exhausted",
|
||||
"elapsed_seconds": 37.05165490099898,
|
||||
"rounds": 6,
|
||||
"tool_calls": 3,
|
||||
"tool_errors": 1,
|
||||
"invalid_actions": 4,
|
||||
"input_tokens": 15948,
|
||||
"output_tokens": 191,
|
||||
"returned_chars": 0,
|
||||
"references": []
|
||||
},
|
||||
{
|
||||
"id": "flask-01",
|
||||
"repository": "flask",
|
||||
"scores": {
|
||||
"file_hit": false,
|
||||
"target_hit": false,
|
||||
"line_precision": 0.0,
|
||||
"line_recall": 0.0,
|
||||
"line_f1": 0.0,
|
||||
"returned_lines": 0,
|
||||
"overlap_lines": 0
|
||||
},
|
||||
"status": "budget_exhausted",
|
||||
"elapsed_seconds": 31.567920669003797,
|
||||
"rounds": 6,
|
||||
"tool_calls": 5,
|
||||
"tool_errors": 1,
|
||||
"invalid_actions": 2,
|
||||
"input_tokens": 19545,
|
||||
"output_tokens": 171,
|
||||
"returned_chars": 0,
|
||||
"references": []
|
||||
},
|
||||
{
|
||||
"id": "flask-02",
|
||||
"repository": "flask",
|
||||
"scores": {
|
||||
"file_hit": false,
|
||||
"target_hit": false,
|
||||
"line_precision": 0.0,
|
||||
"line_recall": 0.0,
|
||||
"line_f1": 0.0,
|
||||
"returned_lines": 0,
|
||||
"overlap_lines": 0
|
||||
},
|
||||
"status": "budget_exhausted",
|
||||
"elapsed_seconds": 31.409248538999236,
|
||||
"rounds": 6,
|
||||
"tool_calls": 5,
|
||||
"tool_errors": 1,
|
||||
"invalid_actions": 2,
|
||||
"input_tokens": 18789,
|
||||
"output_tokens": 171,
|
||||
"returned_chars": 0,
|
||||
"references": []
|
||||
},
|
||||
{
|
||||
"id": "flask-03",
|
||||
"repository": "flask",
|
||||
"scores": {
|
||||
"file_hit": false,
|
||||
"target_hit": false,
|
||||
"line_precision": 0.0,
|
||||
"line_recall": 0.0,
|
||||
"line_f1": 0.0,
|
||||
"returned_lines": 0,
|
||||
"overlap_lines": 0
|
||||
},
|
||||
"status": "budget_exhausted",
|
||||
"elapsed_seconds": 30.19136672300374,
|
||||
"rounds": 6,
|
||||
"tool_calls": 5,
|
||||
"tool_errors": 1,
|
||||
"invalid_actions": 2,
|
||||
"input_tokens": 14973,
|
||||
"output_tokens": 176,
|
||||
"returned_chars": 0,
|
||||
"references": []
|
||||
},
|
||||
{
|
||||
"id": "flask-04",
|
||||
"repository": "flask",
|
||||
"scores": {
|
||||
"file_hit": false,
|
||||
"target_hit": false,
|
||||
"line_precision": 0.0,
|
||||
"line_recall": 0.0,
|
||||
"line_f1": 0.0,
|
||||
"returned_lines": 0,
|
||||
"overlap_lines": 0
|
||||
},
|
||||
"status": "budget_exhausted",
|
||||
"elapsed_seconds": 32.381893327998114,
|
||||
"rounds": 6,
|
||||
"tool_calls": 5,
|
||||
"tool_errors": 1,
|
||||
"invalid_actions": 2,
|
||||
"input_tokens": 18871,
|
||||
"output_tokens": 176,
|
||||
"returned_chars": 0,
|
||||
"references": []
|
||||
},
|
||||
{
|
||||
"id": "flask-05",
|
||||
"repository": "flask",
|
||||
"scores": {
|
||||
"file_hit": false,
|
||||
"target_hit": false,
|
||||
"line_precision": 0.0,
|
||||
"line_recall": 0.0,
|
||||
"line_f1": 0.0,
|
||||
"returned_lines": 0,
|
||||
"overlap_lines": 0
|
||||
},
|
||||
"status": "budget_exhausted",
|
||||
"elapsed_seconds": 34.9670127329955,
|
||||
"rounds": 6,
|
||||
"tool_calls": 3,
|
||||
"tool_errors": 1,
|
||||
"invalid_actions": 4,
|
||||
"input_tokens": 15821,
|
||||
"output_tokens": 199,
|
||||
"returned_chars": 0,
|
||||
"references": []
|
||||
},
|
||||
{
|
||||
"id": "flask-06",
|
||||
"repository": "flask",
|
||||
"scores": {
|
||||
"file_hit": false,
|
||||
"target_hit": false,
|
||||
"line_precision": 0.0,
|
||||
"line_recall": 0.0,
|
||||
"line_f1": 0.0,
|
||||
"returned_lines": 0,
|
||||
"overlap_lines": 0
|
||||
},
|
||||
"status": "budget_exhausted",
|
||||
"elapsed_seconds": 30.41062429500016,
|
||||
"rounds": 6,
|
||||
"tool_calls": 5,
|
||||
"tool_errors": 1,
|
||||
"invalid_actions": 2,
|
||||
"input_tokens": 16521,
|
||||
"output_tokens": 169,
|
||||
"returned_chars": 0,
|
||||
"references": []
|
||||
},
|
||||
{
|
||||
"id": "flask-07",
|
||||
"repository": "flask",
|
||||
"scores": {
|
||||
"file_hit": false,
|
||||
"target_hit": false,
|
||||
"line_precision": 0.0,
|
||||
"line_recall": 0.0,
|
||||
"line_f1": 0.0,
|
||||
"returned_lines": 27,
|
||||
"overlap_lines": 0
|
||||
},
|
||||
"status": "completed",
|
||||
"elapsed_seconds": 15.42695528799959,
|
||||
"rounds": 3,
|
||||
"tool_calls": 3,
|
||||
"tool_errors": 0,
|
||||
"invalid_actions": 0,
|
||||
"input_tokens": 6132,
|
||||
"output_tokens": 92,
|
||||
"returned_chars": 993,
|
||||
"references": [
|
||||
{
|
||||
"path": "docs/debugging.rst",
|
||||
"start_line": 14,
|
||||
"end_line": 40,
|
||||
"sha256": "09bf141a35b5dd6ee45531bb77d9af7bfb94b46c8c8c11f36c26317de07e7b0c",
|
||||
"verified": true
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "flask-08",
|
||||
"repository": "flask",
|
||||
"scores": {
|
||||
"file_hit": true,
|
||||
"target_hit": false,
|
||||
"line_precision": 0.0,
|
||||
"line_recall": 0.0,
|
||||
"line_f1": 0.0,
|
||||
"returned_lines": 32,
|
||||
"overlap_lines": 0
|
||||
},
|
||||
"status": "completed",
|
||||
"elapsed_seconds": 16.30582055800187,
|
||||
"rounds": 3,
|
||||
"tool_calls": 3,
|
||||
"tool_errors": 0,
|
||||
"invalid_actions": 0,
|
||||
"input_tokens": 8562,
|
||||
"output_tokens": 89,
|
||||
"returned_chars": 1396,
|
||||
"references": [
|
||||
{
|
||||
"path": "src/flask/helpers.py",
|
||||
"start_line": 318,
|
||||
"end_line": 349,
|
||||
"sha256": "ac96607bbfc9dbcf09d5442fe7e90d7f8a046b0df7d8f3fc3535b44112016d71",
|
||||
"verified": true
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "flask-09",
|
||||
"repository": "flask",
|
||||
"scores": {
|
||||
"file_hit": false,
|
||||
"target_hit": false,
|
||||
"line_precision": 0.0,
|
||||
"line_recall": 0.0,
|
||||
"line_f1": 0.0,
|
||||
"returned_lines": 0,
|
||||
"overlap_lines": 0
|
||||
},
|
||||
"status": "budget_exhausted",
|
||||
"elapsed_seconds": 34.930573302000994,
|
||||
"rounds": 6,
|
||||
"tool_calls": 5,
|
||||
"tool_errors": 0,
|
||||
"invalid_actions": 2,
|
||||
"input_tokens": 24485,
|
||||
"output_tokens": 168,
|
||||
"returned_chars": 0,
|
||||
"references": []
|
||||
},
|
||||
{
|
||||
"id": "flask-10",
|
||||
"repository": "flask",
|
||||
"scores": {
|
||||
"file_hit": false,
|
||||
"target_hit": false,
|
||||
"line_precision": 0.0,
|
||||
"line_recall": 0.0,
|
||||
"line_f1": 0.0,
|
||||
"returned_lines": 0,
|
||||
"overlap_lines": 0
|
||||
},
|
||||
"status": "budget_exhausted",
|
||||
"elapsed_seconds": 34.84753715899569,
|
||||
"rounds": 6,
|
||||
"tool_calls": 5,
|
||||
"tool_errors": 1,
|
||||
"invalid_actions": 2,
|
||||
"input_tokens": 17346,
|
||||
"output_tokens": 173,
|
||||
"returned_chars": 0,
|
||||
"references": []
|
||||
},
|
||||
{
|
||||
"id": "click-01",
|
||||
"repository": "click",
|
||||
"scores": {
|
||||
"file_hit": false,
|
||||
"target_hit": false,
|
||||
"line_precision": 0.0,
|
||||
"line_recall": 0.0,
|
||||
"line_f1": 0.0,
|
||||
"returned_lines": 0,
|
||||
"overlap_lines": 0
|
||||
},
|
||||
"status": "budget_exhausted",
|
||||
"elapsed_seconds": 36.52237438899465,
|
||||
"rounds": 6,
|
||||
"tool_calls": 5,
|
||||
"tool_errors": 1,
|
||||
"invalid_actions": 2,
|
||||
"input_tokens": 15572,
|
||||
"output_tokens": 170,
|
||||
"returned_chars": 0,
|
||||
"references": []
|
||||
},
|
||||
{
|
||||
"id": "click-02",
|
||||
"repository": "click",
|
||||
"scores": {
|
||||
"file_hit": false,
|
||||
"target_hit": false,
|
||||
"line_precision": 0.0,
|
||||
"line_recall": 0.0,
|
||||
"line_f1": 0.0,
|
||||
"returned_lines": 0,
|
||||
"overlap_lines": 0
|
||||
},
|
||||
"status": "budget_exhausted",
|
||||
"elapsed_seconds": 45.453592761994514,
|
||||
"rounds": 6,
|
||||
"tool_calls": 5,
|
||||
"tool_errors": 1,
|
||||
"invalid_actions": 2,
|
||||
"input_tokens": 17143,
|
||||
"output_tokens": 172,
|
||||
"returned_chars": 0,
|
||||
"references": []
|
||||
},
|
||||
{
|
||||
"id": "click-03",
|
||||
"repository": "click",
|
||||
"scores": {
|
||||
"file_hit": false,
|
||||
"target_hit": false,
|
||||
"line_precision": 0.0,
|
||||
"line_recall": 0.0,
|
||||
"line_f1": 0.0,
|
||||
"returned_lines": 0,
|
||||
"overlap_lines": 0
|
||||
},
|
||||
"status": "budget_exhausted",
|
||||
"elapsed_seconds": 43.93638536000071,
|
||||
"rounds": 6,
|
||||
"tool_calls": 5,
|
||||
"tool_errors": 1,
|
||||
"invalid_actions": 2,
|
||||
"input_tokens": 18195,
|
||||
"output_tokens": 168,
|
||||
"returned_chars": 0,
|
||||
"references": []
|
||||
},
|
||||
{
|
||||
"id": "click-04",
|
||||
"repository": "click",
|
||||
"scores": {
|
||||
"file_hit": false,
|
||||
"target_hit": false,
|
||||
"line_precision": 0.0,
|
||||
"line_recall": 0.0,
|
||||
"line_f1": 0.0,
|
||||
"returned_lines": 0,
|
||||
"overlap_lines": 0
|
||||
},
|
||||
"status": "budget_exhausted",
|
||||
"elapsed_seconds": 43.39338765000139,
|
||||
"rounds": 6,
|
||||
"tool_calls": 5,
|
||||
"tool_errors": 1,
|
||||
"invalid_actions": 2,
|
||||
"input_tokens": 18602,
|
||||
"output_tokens": 169,
|
||||
"returned_chars": 0,
|
||||
"references": []
|
||||
},
|
||||
{
|
||||
"id": "click-05",
|
||||
"repository": "click",
|
||||
"scores": {
|
||||
"file_hit": false,
|
||||
"target_hit": false,
|
||||
"line_precision": 0.0,
|
||||
"line_recall": 0.0,
|
||||
"line_f1": 0.0,
|
||||
"returned_lines": 0,
|
||||
"overlap_lines": 0
|
||||
},
|
||||
"status": "budget_exhausted",
|
||||
"elapsed_seconds": 43.256231248000404,
|
||||
"rounds": 6,
|
||||
"tool_calls": 6,
|
||||
"tool_errors": 1,
|
||||
"invalid_actions": 1,
|
||||
"input_tokens": 19777,
|
||||
"output_tokens": 156,
|
||||
"returned_chars": 0,
|
||||
"references": []
|
||||
},
|
||||
{
|
||||
"id": "click-06",
|
||||
"repository": "click",
|
||||
"scores": {
|
||||
"file_hit": false,
|
||||
"target_hit": false,
|
||||
"line_precision": 0.0,
|
||||
"line_recall": 0.0,
|
||||
"line_f1": 0.0,
|
||||
"returned_lines": 68,
|
||||
"overlap_lines": 0
|
||||
},
|
||||
"status": "completed",
|
||||
"elapsed_seconds": 23.925040923997585,
|
||||
"rounds": 3,
|
||||
"tool_calls": 3,
|
||||
"tool_errors": 0,
|
||||
"invalid_actions": 0,
|
||||
"input_tokens": 8355,
|
||||
"output_tokens": 93,
|
||||
"returned_chars": 2452,
|
||||
"references": [
|
||||
{
|
||||
"path": "src/click/decorators.py",
|
||||
"start_line": 28,
|
||||
"end_line": 95,
|
||||
"sha256": "e4feda6e126d010629fca1e08d4be132fe3ae044703b3aefd9e9cd9279701f24",
|
||||
"verified": true
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "click-07",
|
||||
"repository": "click",
|
||||
"scores": {
|
||||
"file_hit": false,
|
||||
"target_hit": false,
|
||||
"line_precision": 0.0,
|
||||
"line_recall": 0.0,
|
||||
"line_f1": 0.0,
|
||||
"returned_lines": 0,
|
||||
"overlap_lines": 0
|
||||
},
|
||||
"status": "budget_exhausted",
|
||||
"elapsed_seconds": 47.61191895700176,
|
||||
"rounds": 6,
|
||||
"tool_calls": 5,
|
||||
"tool_errors": 1,
|
||||
"invalid_actions": 2,
|
||||
"input_tokens": 19285,
|
||||
"output_tokens": 171,
|
||||
"returned_chars": 0,
|
||||
"references": []
|
||||
},
|
||||
{
|
||||
"id": "click-08",
|
||||
"repository": "click",
|
||||
"scores": {
|
||||
"file_hit": false,
|
||||
"target_hit": false,
|
||||
"line_precision": 0.0,
|
||||
"line_recall": 0.0,
|
||||
"line_f1": 0.0,
|
||||
"returned_lines": 0,
|
||||
"overlap_lines": 0
|
||||
},
|
||||
"status": "budget_exhausted",
|
||||
"elapsed_seconds": 43.185091651997936,
|
||||
"rounds": 6,
|
||||
"tool_calls": 5,
|
||||
"tool_errors": 1,
|
||||
"invalid_actions": 2,
|
||||
"input_tokens": 16813,
|
||||
"output_tokens": 175,
|
||||
"returned_chars": 0,
|
||||
"references": []
|
||||
},
|
||||
{
|
||||
"id": "click-09",
|
||||
"repository": "click",
|
||||
"scores": {
|
||||
"file_hit": false,
|
||||
"target_hit": false,
|
||||
"line_precision": 0.0,
|
||||
"line_recall": 0.0,
|
||||
"line_f1": 0.0,
|
||||
"returned_lines": 0,
|
||||
"overlap_lines": 0
|
||||
},
|
||||
"status": "budget_exhausted",
|
||||
"elapsed_seconds": 34.059470203000274,
|
||||
"rounds": 6,
|
||||
"tool_calls": 5,
|
||||
"tool_errors": 1,
|
||||
"invalid_actions": 2,
|
||||
"input_tokens": 18742,
|
||||
"output_tokens": 177,
|
||||
"returned_chars": 0,
|
||||
"references": []
|
||||
},
|
||||
{
|
||||
"id": "click-10",
|
||||
"repository": "click",
|
||||
"scores": {
|
||||
"file_hit": false,
|
||||
"target_hit": false,
|
||||
"line_precision": 0.0,
|
||||
"line_recall": 0.0,
|
||||
"line_f1": 0.0,
|
||||
"returned_lines": 0,
|
||||
"overlap_lines": 0
|
||||
},
|
||||
"status": "budget_exhausted",
|
||||
"elapsed_seconds": 38.496093912000106,
|
||||
"rounds": 6,
|
||||
"tool_calls": 3,
|
||||
"tool_errors": 1,
|
||||
"invalid_actions": 4,
|
||||
"input_tokens": 15509,
|
||||
"output_tokens": 201,
|
||||
"returned_chars": 0,
|
||||
"references": []
|
||||
}
|
||||
]
|
||||
@@ -0,0 +1,525 @@
|
||||
{
|
||||
"suite": {
|
||||
"name": "live-search-v1",
|
||||
"scope": "30 hand-authored English single-function localization tasks in three public Python repositories. Development benchmark, not a downstream coding-task or contamination-free evaluation.",
|
||||
"repositories": {
|
||||
"requests": {
|
||||
"commit": "b25c87d7cb8d6a18a37fa12442b5f883f9e41741",
|
||||
"url": "https://github.com/psf/requests.git"
|
||||
},
|
||||
"flask": {
|
||||
"commit": "2c1b30d0503cfb064f1cb252e6614a06915a362a",
|
||||
"url": "https://github.com/pallets/flask.git"
|
||||
},
|
||||
"click": {
|
||||
"commit": "fd183b2ced1cb5857784fe7fb22f4982f671f098",
|
||||
"url": "https://github.com/pallets/click.git"
|
||||
}
|
||||
},
|
||||
"tasks": [
|
||||
{
|
||||
"id": "requests-01",
|
||||
"repository": "requests",
|
||||
"query": "Find where an unsuccessful HTTP status becomes an exception containing the server reason and URL.",
|
||||
"targets": [
|
||||
{
|
||||
"path": "src/requests/models.py",
|
||||
"symbol": "Response.raise_for_status",
|
||||
"start_line": 1002,
|
||||
"end_line": 1026,
|
||||
"sha256": "32365d67893bb67c3ed67cf93ca4a18e63e6ab29342fa0dc8b09c59e06ff564e"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "requests-02",
|
||||
"repository": "requests",
|
||||
"query": "Locate the response iterator that preserves an incomplete trailing line across downloaded chunks.",
|
||||
"targets": [
|
||||
{
|
||||
"path": "src/requests/models.py",
|
||||
"symbol": "Response.iter_lines",
|
||||
"start_line": 867,
|
||||
"end_line": 888,
|
||||
"sha256": "32365d67893bb67c3ed67cf93ca4a18e63e6ab29342fa0dc8b09c59e06ff564e"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "requests-03",
|
||||
"repository": "requests",
|
||||
"query": "Where is it decided whether credentials may survive a redirect to another hostname, port, or protocol?",
|
||||
"targets": [
|
||||
{
|
||||
"path": "src/requests/sessions.py",
|
||||
"symbol": "SessionRedirectMixin.should_strip_auth",
|
||||
"start_line": 129,
|
||||
"end_line": 157,
|
||||
"sha256": "0a5d5da449ce7f0af3ccf6e4bbe7a67a935e37846dff4ff9f08cb6c7e2464e6f"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "requests-04",
|
||||
"repository": "requests",
|
||||
"query": "Find the logic that changes the HTTP verb when following 301, 302, or 303 redirects.",
|
||||
"targets": [
|
||||
{
|
||||
"path": "src/requests/sessions.py",
|
||||
"symbol": "SessionRedirectMixin.rebuild_method",
|
||||
"start_line": 337,
|
||||
"end_line": 353,
|
||||
"sha256": "0a5d5da449ce7f0af3ccf6e4bbe7a67a935e37846dff4ff9f08cb6c7e2464e6f"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "requests-05",
|
||||
"repository": "requests",
|
||||
"query": "Locate where environment proxy settings and certificate bundle variables are merged with session options.",
|
||||
"targets": [
|
||||
{
|
||||
"path": "src/requests/sessions.py",
|
||||
"symbol": "Session.merge_environment_settings",
|
||||
"start_line": 757,
|
||||
"end_line": 779,
|
||||
"sha256": "0a5d5da449ce7f0af3ccf6e4bbe7a67a935e37846dff4ff9f08cb6c7e2464e6f"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "requests-06",
|
||||
"repository": "requests",
|
||||
"query": "Find where a 401 digest challenge causes the original request to be copied and sent again with authentication.",
|
||||
"targets": [
|
||||
{
|
||||
"path": "src/requests/auth.py",
|
||||
"symbol": "HTTPDigestAuth.handle_401",
|
||||
"start_line": 250,
|
||||
"end_line": 283,
|
||||
"sha256": "905ef9b6a9cb72d67d31ffe19bd4d9223e1c4169cde6ec51cfca16b31e70991d"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "requests-07",
|
||||
"repository": "requests",
|
||||
"query": "Locate the helper that fills a cookie container from a mapping while optionally preserving existing names.",
|
||||
"targets": [
|
||||
{
|
||||
"path": "src/requests/cookies.py",
|
||||
"symbol": "cookiejar_from_dict",
|
||||
"start_line": 530,
|
||||
"end_line": 539,
|
||||
"sha256": "6cd8be8aa123e0d3d9d34fa86feac7bf392f39bccdde5129830de0ea9692dd7c"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "requests-08",
|
||||
"repository": "requests",
|
||||
"query": "Find where credentials are loaded from the users netrc file, including NETRC and home-directory lookup.",
|
||||
"targets": [
|
||||
{
|
||||
"path": "src/requests/utils.py",
|
||||
"symbol": "get_netrc_auth",
|
||||
"start_line": 210,
|
||||
"end_line": 248,
|
||||
"sha256": "5aa53ceab677c2f842fad42359c8ed1ff1c4299c1607789609957a496e4311d4"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "requests-09",
|
||||
"repository": "requests",
|
||||
"query": "Find the implementation that seeks a request body back to its saved position and fails for an unrewindable stream.",
|
||||
"targets": [
|
||||
{
|
||||
"path": "src/requests/utils.py",
|
||||
"symbol": "rewind_body",
|
||||
"start_line": 1075,
|
||||
"end_line": 1086,
|
||||
"sha256": "5aa53ceab677c2f842fad42359c8ed1ff1c4299c1607789609957a496e4311d4"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "requests-10",
|
||||
"repository": "requests",
|
||||
"query": "Where is a request URL validated, its international hostname encoded, and its query parameters appended?",
|
||||
"targets": [
|
||||
{
|
||||
"path": "src/requests/models.py",
|
||||
"symbol": "PreparedRequest.prepare_url",
|
||||
"start_line": 416,
|
||||
"end_line": 481,
|
||||
"sha256": "32365d67893bb67c3ed67cf93ca4a18e63e6ab29342fa0dc8b09c59e06ff564e"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "flask-01",
|
||||
"repository": "flask",
|
||||
"query": "Find the code that invokes the view function selected by the matched URL rule.",
|
||||
"targets": [
|
||||
{
|
||||
"path": "src/flask/app.py",
|
||||
"symbol": "Flask.dispatch_request",
|
||||
"start_line": 889,
|
||||
"end_line": 902,
|
||||
"sha256": "5c6aa0151b0b8018732280761d27eda0c4c83378630e21faadd57b73783720a7"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "flask-02",
|
||||
"repository": "flask",
|
||||
"query": "Where are view return values such as tuples, dictionaries and strings converted into a response object?",
|
||||
"targets": [
|
||||
{
|
||||
"path": "src/flask/app.py",
|
||||
"symbol": "Flask.make_response",
|
||||
"start_line": 1186,
|
||||
"end_line": 1269,
|
||||
"sha256": "5c6aa0151b0b8018732280761d27eda0c4c83378630e21faadd57b73783720a7"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "flask-03",
|
||||
"repository": "flask",
|
||||
"query": "Find where coroutine view functions are adapted for synchronous request handling.",
|
||||
"targets": [
|
||||
{
|
||||
"path": "src/flask/app.py",
|
||||
"symbol": "Flask.ensure_sync",
|
||||
"start_line": 975,
|
||||
"end_line": 978,
|
||||
"sha256": "5c6aa0151b0b8018732280761d27eda0c4c83378630e21faadd57b73783720a7"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "flask-04",
|
||||
"repository": "flask",
|
||||
"query": "Locate the processing that runs after-request callbacks and saves the session before returning the response.",
|
||||
"targets": [
|
||||
{
|
||||
"path": "src/flask/app.py",
|
||||
"symbol": "Flask.process_response",
|
||||
"start_line": 1311,
|
||||
"end_line": 1324,
|
||||
"sha256": "5c6aa0151b0b8018732280761d27eda0c4c83378630e21faadd57b73783720a7"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "flask-05",
|
||||
"repository": "flask",
|
||||
"query": "Find where before-request handlers can short-circuit normal request dispatch by returning a value.",
|
||||
"targets": [
|
||||
{
|
||||
"path": "src/flask/app.py",
|
||||
"symbol": "Flask.preprocess_request",
|
||||
"start_line": 1281,
|
||||
"end_line": 1296,
|
||||
"sha256": "5c6aa0151b0b8018732280761d27eda0c4c83378630e21faadd57b73783720a7"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "flask-06",
|
||||
"repository": "flask",
|
||||
"query": "Where is a streaming generator wrapped so the request context remains active while producing its items?",
|
||||
"targets": [
|
||||
{
|
||||
"path": "src/flask/helpers.py",
|
||||
"symbol": "stream_with_context",
|
||||
"start_line": 104,
|
||||
"end_line": 143,
|
||||
"sha256": "ac96607bbfc9dbcf09d5442fe7e90d7f8a046b0df7d8f3fc3535b44112016d71"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "flask-07",
|
||||
"repository": "flask",
|
||||
"query": "Find where a categorized one-time message is stored in the session and a notification signal is emitted.",
|
||||
"targets": [
|
||||
{
|
||||
"path": "src/flask/helpers.py",
|
||||
"symbol": "flash",
|
||||
"start_line": 340,
|
||||
"end_line": 349,
|
||||
"sha256": "ac96607bbfc9dbcf09d5442fe7e90d7f8a046b0df7d8f3fc3535b44112016d71"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "flask-08",
|
||||
"repository": "flask",
|
||||
"query": "Locate where stored one-time messages are popped from the session, cached for the request, and filtered by category.",
|
||||
"targets": [
|
||||
{
|
||||
"path": "src/flask/helpers.py",
|
||||
"symbol": "get_flashed_messages",
|
||||
"start_line": 383,
|
||||
"end_line": 391,
|
||||
"sha256": "ac96607bbfc9dbcf09d5442fe7e90d7f8a046b0df7d8f3fc3535b44112016d71"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "flask-09",
|
||||
"repository": "flask",
|
||||
"query": "Find where the command-line startup reads dotenv files without replacing environment variables already set.",
|
||||
"targets": [
|
||||
{
|
||||
"path": "src/flask/cli.py",
|
||||
"symbol": "load_dotenv",
|
||||
"start_line": 740,
|
||||
"end_line": 771,
|
||||
"sha256": "3df87bdbe07196fa07d101c20ce7351ef5c6ecaab95deb5fdf3ccdcf690d4879"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "flask-10",
|
||||
"repository": "flask",
|
||||
"query": "Locate where a signed session cookie is verified and an invalid signature produces an empty session.",
|
||||
"targets": [
|
||||
{
|
||||
"path": "src/flask/sessions.py",
|
||||
"symbol": "SecureCookieSessionInterface.open_session",
|
||||
"start_line": 338,
|
||||
"end_line": 349,
|
||||
"sha256": "76ebd81a608687f1f772032ecb6a1e4a3ac2b02bcbca517808128e5e9374426b"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "click-01",
|
||||
"repository": "click",
|
||||
"query": "Find where a supplied option value is normalized and matched against the allowed choices.",
|
||||
"targets": [
|
||||
{
|
||||
"path": "src/click/types.py",
|
||||
"symbol": "Choice.convert",
|
||||
"start_line": 344,
|
||||
"end_line": 358,
|
||||
"sha256": "2814d1c4ddbc711d5567999bf62257f7c3104b0fe9f521b3963a9f108f33e53c"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "click-02",
|
||||
"repository": "click",
|
||||
"query": "Locate where a filesystem argument is checked for existence, readability, writability, and allowed file or directory type.",
|
||||
"targets": [
|
||||
{
|
||||
"path": "src/click/types.py",
|
||||
"symbol": "Path.convert",
|
||||
"start_line": 930,
|
||||
"end_line": 995,
|
||||
"sha256": "2814d1c4ddbc711d5567999bf62257f7c3104b0fe9f521b3963a9f108f33e53c"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "click-03",
|
||||
"repository": "click",
|
||||
"query": "Find where a date argument is parsed by trying several accepted formats and reports a failure if none match.",
|
||||
"targets": [
|
||||
{
|
||||
"path": "src/click/types.py",
|
||||
"symbol": "DateTime.convert",
|
||||
"start_line": 448,
|
||||
"end_line": 466,
|
||||
"sha256": "2814d1c4ddbc711d5567999bf62257f7c3104b0fe9f521b3963a9f108f33e53c"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "click-04",
|
||||
"repository": "click",
|
||||
"query": "Where is an option value from an environment variable split and grouped for multiple arguments?",
|
||||
"targets": [
|
||||
{
|
||||
"path": "src/click/core.py",
|
||||
"symbol": "Option.value_from_envvar",
|
||||
"start_line": 2983,
|
||||
"end_line": 2996,
|
||||
"sha256": "814869352f5c14119d1175dd8ac1951be791bc67f8f514f215a831ba5ab0745c"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "click-05",
|
||||
"repository": "click",
|
||||
"query": "Locate where a command group resolves a subcommand name, retries normalized names, and rejects unknown commands.",
|
||||
"targets": [
|
||||
{
|
||||
"path": "src/click/core.py",
|
||||
"symbol": "Group.resolve_command",
|
||||
"start_line": 1867,
|
||||
"end_line": 1889,
|
||||
"sha256": "814869352f5c14119d1175dd8ac1951be791bc67f8f514f215a831ba5ab0745c"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "click-06",
|
||||
"repository": "click",
|
||||
"query": "Find the helper that enters a context manager and registers its cleanup with the command context.",
|
||||
"targets": [
|
||||
{
|
||||
"path": "src/click/core.py",
|
||||
"symbol": "Context.with_resource",
|
||||
"start_line": 598,
|
||||
"end_line": 598,
|
||||
"sha256": "814869352f5c14119d1175dd8ac1951be791bc67f8f514f215a831ba5ab0745c"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "click-07",
|
||||
"repository": "click",
|
||||
"query": "Find where each element of a tuple argument is converted using its corresponding parameter type.",
|
||||
"targets": [
|
||||
{
|
||||
"path": "src/click/types.py",
|
||||
"symbol": "Tuple.convert",
|
||||
"start_line": 1049,
|
||||
"end_line": 1065,
|
||||
"sha256": "2814d1c4ddbc711d5567999bf62257f7c3104b0fe9f521b3963a9f108f33e53c"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "click-08",
|
||||
"repository": "click",
|
||||
"query": "Locate the interactive yes-or-no question loop that retries invalid answers and can abort on rejection.",
|
||||
"targets": [
|
||||
{
|
||||
"path": "src/click/termui.py",
|
||||
"symbol": "confirm",
|
||||
"start_line": 223,
|
||||
"end_line": 252,
|
||||
"sha256": "bc062b282d9aedffcd7c42210138445587dafff89be5741b4d2086b499400510"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "click-09",
|
||||
"repository": "click",
|
||||
"query": "Find the testing helper that temporarily changes the current directory and removes its temporary directory on exit.",
|
||||
"targets": [
|
||||
{
|
||||
"path": "src/click/testing.py",
|
||||
"symbol": "CliRunner.isolated_filesystem",
|
||||
"start_line": 552,
|
||||
"end_line": 565,
|
||||
"sha256": "d9e2dd01a0890864e83f94f0f9737c263ac36f7c1b2487c047183777472b5b93"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "click-10",
|
||||
"repository": "click",
|
||||
"query": "Where does a lazily opened file acquire its actual stream and translate operating-system errors into a file error?",
|
||||
"targets": [
|
||||
{
|
||||
"path": "src/click/utils.py",
|
||||
"symbol": "LazyFile.open",
|
||||
"start_line": 156,
|
||||
"end_line": 167,
|
||||
"sha256": "6f5326faeb040c11ed13070f96d3c89f7cd22b89f0d8a4e9e460bbfe849394ba"
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
},
|
||||
"suite_sha256": "d06effdf41ee2e38bf8f44629949d9b4491067986fc5a73aef22cf80c5a5bf96",
|
||||
"model": {
|
||||
"backend": "keyword",
|
||||
"model": null,
|
||||
"max_query_terms": 12,
|
||||
"window_radius_lines": 12,
|
||||
"max_read_calls": 3,
|
||||
"ranking": "distinct query terms in matching lines within each window",
|
||||
"stopwords": [
|
||||
"after",
|
||||
"and",
|
||||
"are",
|
||||
"before",
|
||||
"code",
|
||||
"containing",
|
||||
"current",
|
||||
"does",
|
||||
"find",
|
||||
"for",
|
||||
"from",
|
||||
"function",
|
||||
"given",
|
||||
"handles",
|
||||
"handling",
|
||||
"how",
|
||||
"implementation",
|
||||
"into",
|
||||
"locate",
|
||||
"method",
|
||||
"repository",
|
||||
"return",
|
||||
"returns",
|
||||
"source",
|
||||
"that",
|
||||
"the",
|
||||
"this",
|
||||
"used",
|
||||
"using",
|
||||
"what",
|
||||
"when",
|
||||
"where",
|
||||
"which",
|
||||
"with"
|
||||
]
|
||||
},
|
||||
"source_sha256": {
|
||||
"__init__.py": "dcd2b573883b8068e806e3052adf9c03728f1cf621d35ff013179de76f76c702",
|
||||
"text.py": "8db7b80ee446480175f7691872303c8b72fd014604ad4fb575b359cee71f1938",
|
||||
"io.py": "b29c21db767ea9773281b17fe0754375153d18fb64f314907528f00d584a047f",
|
||||
"data.py": "20a891eff11d555d2301b62f4953146603efc87b03da4802f136f694ebdba78d",
|
||||
"encoder.py": "cb5671c08f28a70ffd0e2a7033528efc3581896a27242d55eb250ce922ab038f",
|
||||
"metrics.py": "cb93d30878c0c1006f06d1d0c4cbb188f90c6f4fc65f69f9a3b11d847f28ad27",
|
||||
"train.py": "cf20f6189967cfc88f3849bc3a225f2680c42026d53f204d9d5afc05fe88a2d6",
|
||||
"lexical.py": "6c47c834c6bbf7b317e2514057eee7444720581f5ec452ecb01ac666397b86b3",
|
||||
"symbols.py": "a377d6aa2c6b479c2ee115dd8e4859f0c06d188ae97b86fa1c7133885ee32d86",
|
||||
"index.py": "e8b0a666b41fed48d98616d5dcdd104e59c4308ac990281b463e5ee68f97877f",
|
||||
"evaluate.py": "29c942b9cbfff21ba8514511e27752527f0ec5e2cad5fbf57ec3b220fc7acd0b",
|
||||
"__main__.py": "ab30d9b696e41ece67094e19c87b1a31fc0ebce8d3afa79dc8eaf561f8fb4895",
|
||||
"download.py": "24f9802fb7cc82b81f6b1cb93e9244352d7b86feeb0a54ec055fb3e8d0a6a4d7",
|
||||
"cli.py": "d081962fc459808c65bd4c1d389e12ad9a9466df0a1374c8095a8bd5ccbb369e",
|
||||
"scout.py": "c3150baae51f37646305fee1d2c61e7b37f2ec9c4a0ba7f92ab4d7d6b1710161",
|
||||
"server.py": "483fe8051012249e38a6836cbe217d8c581d033e4d19026a29a7bd5c83878190",
|
||||
"live_tools.py": "e10fb6aaeaa6d7c4a5a35c0c6a210be4a4c283b18d215767d6bee0bac707d14c",
|
||||
"local_policy.py": "b0711b46a6399dfbeeecdcc045fb7392030e187d7e21c6e4fa9b51e6fc8056f4",
|
||||
"agent.py": "47357a900e9696b7ec5268eb031067eb0069faaeef92fd3159bd822d6bc67cdb",
|
||||
"live_server.py": "ccfb3c63115fe991c2f5f22c05f794d89c90d783b52f8ddbb668a8541e3af6fe",
|
||||
"native_protocol.py": "3a4d16c1d2cd90cb7d81e635f40d721270b0d541f454557a6d14e80b6b1073a7",
|
||||
"eval_live.py": "38250d45fd7e60c23c4c86f5819242736f29df8af384f92713128c84ffd734ff",
|
||||
"prepare_live.py": "0675173c5e9c116c995c271dc06de914388f19eb83779c07e3009000a810290b",
|
||||
"live_data.py": "fd0bd5b118b2437146e08ef921dc4aa8a1da24e69bcbdd3efc5ff6e9c4fe8a78",
|
||||
"transformers_policy.py": "e8e8a81b70eb37563973e640534b425a7ea9fc7d398a39dc8d8f71b9be3f1b89",
|
||||
"train_policy.py": "803f45cb78087d1899c59bc09b6589e2ac84753c8f4be9938d837bbb5db3def8",
|
||||
"keyword_baseline.py": "a97336c72706d55f190ca3833004c849d365916078566f4d84e6f3d3dd3a04e2"
|
||||
},
|
||||
"max_rounds": null,
|
||||
"max_chars": 6000,
|
||||
"timeout_seconds": null,
|
||||
"context": 0,
|
||||
"max_generation_tokens": 0,
|
||||
"tokenizer": null,
|
||||
"temperature": 0,
|
||||
"seed": 42,
|
||||
"scope": "Single-function localization; no large-model baseline; public repositories may have appeared in the base model's pretraining."
|
||||
}
|
||||
@@ -0,0 +1,84 @@
|
||||
{
|
||||
"tasks": 30,
|
||||
"target_hit_rate": 0,
|
||||
"file_hit_rate": 0.26666666666666666,
|
||||
"macro_line_precision": 0.0,
|
||||
"macro_line_recall": 0.0,
|
||||
"macro_line_f1": 0.0,
|
||||
"latency_median_seconds": 0.048612095499265706,
|
||||
"latency_p95_seconds": 0.06725892820031731,
|
||||
"statuses": {
|
||||
"completed": 30
|
||||
},
|
||||
"total_tool_errors": 0,
|
||||
"total_invalid_actions": 0,
|
||||
"mean_rounds": 0,
|
||||
"mean_tool_calls": 4,
|
||||
"total_input_tokens": 0,
|
||||
"total_output_tokens": 0,
|
||||
"mean_returned_chars": 2521.4666666666667,
|
||||
"per_repository": {
|
||||
"requests": {
|
||||
"tasks": 10,
|
||||
"target_hit_rate": 0,
|
||||
"file_hit_rate": 0.1,
|
||||
"macro_line_precision": 0.0,
|
||||
"macro_line_recall": 0.0,
|
||||
"macro_line_f1": 0.0,
|
||||
"latency_median_seconds": 0.04749804249513545,
|
||||
"latency_p95_seconds": 0.05884885605155432,
|
||||
"statuses": {
|
||||
"completed": 10
|
||||
},
|
||||
"total_tool_errors": 0,
|
||||
"total_invalid_actions": 0,
|
||||
"mean_rounds": 0,
|
||||
"mean_tool_calls": 4,
|
||||
"total_input_tokens": 0,
|
||||
"total_output_tokens": 0,
|
||||
"mean_returned_chars": 2325.2
|
||||
},
|
||||
"flask": {
|
||||
"tasks": 10,
|
||||
"target_hit_rate": 0,
|
||||
"file_hit_rate": 0.5,
|
||||
"macro_line_precision": 0.0,
|
||||
"macro_line_recall": 0.0,
|
||||
"macro_line_f1": 0.0,
|
||||
"latency_median_seconds": 0.055962577996979235,
|
||||
"latency_p95_seconds": 0.06988518085126998,
|
||||
"statuses": {
|
||||
"completed": 10
|
||||
},
|
||||
"total_tool_errors": 0,
|
||||
"total_invalid_actions": 0,
|
||||
"mean_rounds": 0,
|
||||
"mean_tool_calls": 4,
|
||||
"total_input_tokens": 0,
|
||||
"total_output_tokens": 0,
|
||||
"mean_returned_chars": 2740.9
|
||||
},
|
||||
"click": {
|
||||
"tasks": 10,
|
||||
"target_hit_rate": 0,
|
||||
"file_hit_rate": 0.2,
|
||||
"macro_line_precision": 0.0,
|
||||
"macro_line_recall": 0.0,
|
||||
"macro_line_f1": 0.0,
|
||||
"latency_median_seconds": 0.04842947550059762,
|
||||
"latency_p95_seconds": 0.05770411535195308,
|
||||
"statuses": {
|
||||
"completed": 10
|
||||
},
|
||||
"total_tool_errors": 0,
|
||||
"total_invalid_actions": 0,
|
||||
"mean_rounds": 0,
|
||||
"mean_tool_calls": 4,
|
||||
"total_input_tokens": 0,
|
||||
"total_output_tokens": 0,
|
||||
"mean_returned_chars": 2498.3
|
||||
}
|
||||
},
|
||||
"sampled_peak_gpu_memory_mib": null,
|
||||
"gpu_sampling_interval_seconds": 1
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,526 @@
|
||||
{
|
||||
"suite": {
|
||||
"name": "live-search-v1",
|
||||
"scope": "30 hand-authored English single-function localization tasks in three public Python repositories. Development benchmark, not a downstream coding-task or contamination-free evaluation.",
|
||||
"repositories": {
|
||||
"requests": {
|
||||
"commit": "b25c87d7cb8d6a18a37fa12442b5f883f9e41741",
|
||||
"url": "https://github.com/psf/requests.git"
|
||||
},
|
||||
"flask": {
|
||||
"commit": "2c1b30d0503cfb064f1cb252e6614a06915a362a",
|
||||
"url": "https://github.com/pallets/flask.git"
|
||||
},
|
||||
"click": {
|
||||
"commit": "fd183b2ced1cb5857784fe7fb22f4982f671f098",
|
||||
"url": "https://github.com/pallets/click.git"
|
||||
}
|
||||
},
|
||||
"tasks": [
|
||||
{
|
||||
"id": "requests-01",
|
||||
"repository": "requests",
|
||||
"query": "Find where an unsuccessful HTTP status becomes an exception containing the server reason and URL.",
|
||||
"targets": [
|
||||
{
|
||||
"path": "src/requests/models.py",
|
||||
"symbol": "Response.raise_for_status",
|
||||
"start_line": 1002,
|
||||
"end_line": 1026,
|
||||
"sha256": "32365d67893bb67c3ed67cf93ca4a18e63e6ab29342fa0dc8b09c59e06ff564e"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "requests-02",
|
||||
"repository": "requests",
|
||||
"query": "Locate the response iterator that preserves an incomplete trailing line across downloaded chunks.",
|
||||
"targets": [
|
||||
{
|
||||
"path": "src/requests/models.py",
|
||||
"symbol": "Response.iter_lines",
|
||||
"start_line": 867,
|
||||
"end_line": 888,
|
||||
"sha256": "32365d67893bb67c3ed67cf93ca4a18e63e6ab29342fa0dc8b09c59e06ff564e"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "requests-03",
|
||||
"repository": "requests",
|
||||
"query": "Where is it decided whether credentials may survive a redirect to another hostname, port, or protocol?",
|
||||
"targets": [
|
||||
{
|
||||
"path": "src/requests/sessions.py",
|
||||
"symbol": "SessionRedirectMixin.should_strip_auth",
|
||||
"start_line": 129,
|
||||
"end_line": 157,
|
||||
"sha256": "0a5d5da449ce7f0af3ccf6e4bbe7a67a935e37846dff4ff9f08cb6c7e2464e6f"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "requests-04",
|
||||
"repository": "requests",
|
||||
"query": "Find the logic that changes the HTTP verb when following 301, 302, or 303 redirects.",
|
||||
"targets": [
|
||||
{
|
||||
"path": "src/requests/sessions.py",
|
||||
"symbol": "SessionRedirectMixin.rebuild_method",
|
||||
"start_line": 337,
|
||||
"end_line": 353,
|
||||
"sha256": "0a5d5da449ce7f0af3ccf6e4bbe7a67a935e37846dff4ff9f08cb6c7e2464e6f"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "requests-05",
|
||||
"repository": "requests",
|
||||
"query": "Locate where environment proxy settings and certificate bundle variables are merged with session options.",
|
||||
"targets": [
|
||||
{
|
||||
"path": "src/requests/sessions.py",
|
||||
"symbol": "Session.merge_environment_settings",
|
||||
"start_line": 757,
|
||||
"end_line": 779,
|
||||
"sha256": "0a5d5da449ce7f0af3ccf6e4bbe7a67a935e37846dff4ff9f08cb6c7e2464e6f"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "requests-06",
|
||||
"repository": "requests",
|
||||
"query": "Find where a 401 digest challenge causes the original request to be copied and sent again with authentication.",
|
||||
"targets": [
|
||||
{
|
||||
"path": "src/requests/auth.py",
|
||||
"symbol": "HTTPDigestAuth.handle_401",
|
||||
"start_line": 250,
|
||||
"end_line": 283,
|
||||
"sha256": "905ef9b6a9cb72d67d31ffe19bd4d9223e1c4169cde6ec51cfca16b31e70991d"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "requests-07",
|
||||
"repository": "requests",
|
||||
"query": "Locate the helper that fills a cookie container from a mapping while optionally preserving existing names.",
|
||||
"targets": [
|
||||
{
|
||||
"path": "src/requests/cookies.py",
|
||||
"symbol": "cookiejar_from_dict",
|
||||
"start_line": 530,
|
||||
"end_line": 539,
|
||||
"sha256": "6cd8be8aa123e0d3d9d34fa86feac7bf392f39bccdde5129830de0ea9692dd7c"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "requests-08",
|
||||
"repository": "requests",
|
||||
"query": "Find where credentials are loaded from the users netrc file, including NETRC and home-directory lookup.",
|
||||
"targets": [
|
||||
{
|
||||
"path": "src/requests/utils.py",
|
||||
"symbol": "get_netrc_auth",
|
||||
"start_line": 210,
|
||||
"end_line": 248,
|
||||
"sha256": "5aa53ceab677c2f842fad42359c8ed1ff1c4299c1607789609957a496e4311d4"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "requests-09",
|
||||
"repository": "requests",
|
||||
"query": "Find the implementation that seeks a request body back to its saved position and fails for an unrewindable stream.",
|
||||
"targets": [
|
||||
{
|
||||
"path": "src/requests/utils.py",
|
||||
"symbol": "rewind_body",
|
||||
"start_line": 1075,
|
||||
"end_line": 1086,
|
||||
"sha256": "5aa53ceab677c2f842fad42359c8ed1ff1c4299c1607789609957a496e4311d4"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "requests-10",
|
||||
"repository": "requests",
|
||||
"query": "Where is a request URL validated, its international hostname encoded, and its query parameters appended?",
|
||||
"targets": [
|
||||
{
|
||||
"path": "src/requests/models.py",
|
||||
"symbol": "PreparedRequest.prepare_url",
|
||||
"start_line": 416,
|
||||
"end_line": 481,
|
||||
"sha256": "32365d67893bb67c3ed67cf93ca4a18e63e6ab29342fa0dc8b09c59e06ff564e"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "flask-01",
|
||||
"repository": "flask",
|
||||
"query": "Find the code that invokes the view function selected by the matched URL rule.",
|
||||
"targets": [
|
||||
{
|
||||
"path": "src/flask/app.py",
|
||||
"symbol": "Flask.dispatch_request",
|
||||
"start_line": 889,
|
||||
"end_line": 902,
|
||||
"sha256": "5c6aa0151b0b8018732280761d27eda0c4c83378630e21faadd57b73783720a7"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "flask-02",
|
||||
"repository": "flask",
|
||||
"query": "Where are view return values such as tuples, dictionaries and strings converted into a response object?",
|
||||
"targets": [
|
||||
{
|
||||
"path": "src/flask/app.py",
|
||||
"symbol": "Flask.make_response",
|
||||
"start_line": 1186,
|
||||
"end_line": 1269,
|
||||
"sha256": "5c6aa0151b0b8018732280761d27eda0c4c83378630e21faadd57b73783720a7"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "flask-03",
|
||||
"repository": "flask",
|
||||
"query": "Find where coroutine view functions are adapted for synchronous request handling.",
|
||||
"targets": [
|
||||
{
|
||||
"path": "src/flask/app.py",
|
||||
"symbol": "Flask.ensure_sync",
|
||||
"start_line": 975,
|
||||
"end_line": 978,
|
||||
"sha256": "5c6aa0151b0b8018732280761d27eda0c4c83378630e21faadd57b73783720a7"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "flask-04",
|
||||
"repository": "flask",
|
||||
"query": "Locate the processing that runs after-request callbacks and saves the session before returning the response.",
|
||||
"targets": [
|
||||
{
|
||||
"path": "src/flask/app.py",
|
||||
"symbol": "Flask.process_response",
|
||||
"start_line": 1311,
|
||||
"end_line": 1324,
|
||||
"sha256": "5c6aa0151b0b8018732280761d27eda0c4c83378630e21faadd57b73783720a7"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "flask-05",
|
||||
"repository": "flask",
|
||||
"query": "Find where before-request handlers can short-circuit normal request dispatch by returning a value.",
|
||||
"targets": [
|
||||
{
|
||||
"path": "src/flask/app.py",
|
||||
"symbol": "Flask.preprocess_request",
|
||||
"start_line": 1281,
|
||||
"end_line": 1296,
|
||||
"sha256": "5c6aa0151b0b8018732280761d27eda0c4c83378630e21faadd57b73783720a7"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "flask-06",
|
||||
"repository": "flask",
|
||||
"query": "Where is a streaming generator wrapped so the request context remains active while producing its items?",
|
||||
"targets": [
|
||||
{
|
||||
"path": "src/flask/helpers.py",
|
||||
"symbol": "stream_with_context",
|
||||
"start_line": 104,
|
||||
"end_line": 143,
|
||||
"sha256": "ac96607bbfc9dbcf09d5442fe7e90d7f8a046b0df7d8f3fc3535b44112016d71"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "flask-07",
|
||||
"repository": "flask",
|
||||
"query": "Find where a categorized one-time message is stored in the session and a notification signal is emitted.",
|
||||
"targets": [
|
||||
{
|
||||
"path": "src/flask/helpers.py",
|
||||
"symbol": "flash",
|
||||
"start_line": 340,
|
||||
"end_line": 349,
|
||||
"sha256": "ac96607bbfc9dbcf09d5442fe7e90d7f8a046b0df7d8f3fc3535b44112016d71"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "flask-08",
|
||||
"repository": "flask",
|
||||
"query": "Locate where stored one-time messages are popped from the session, cached for the request, and filtered by category.",
|
||||
"targets": [
|
||||
{
|
||||
"path": "src/flask/helpers.py",
|
||||
"symbol": "get_flashed_messages",
|
||||
"start_line": 383,
|
||||
"end_line": 391,
|
||||
"sha256": "ac96607bbfc9dbcf09d5442fe7e90d7f8a046b0df7d8f3fc3535b44112016d71"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "flask-09",
|
||||
"repository": "flask",
|
||||
"query": "Find where the command-line startup reads dotenv files without replacing environment variables already set.",
|
||||
"targets": [
|
||||
{
|
||||
"path": "src/flask/cli.py",
|
||||
"symbol": "load_dotenv",
|
||||
"start_line": 740,
|
||||
"end_line": 771,
|
||||
"sha256": "3df87bdbe07196fa07d101c20ce7351ef5c6ecaab95deb5fdf3ccdcf690d4879"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "flask-10",
|
||||
"repository": "flask",
|
||||
"query": "Locate where a signed session cookie is verified and an invalid signature produces an empty session.",
|
||||
"targets": [
|
||||
{
|
||||
"path": "src/flask/sessions.py",
|
||||
"symbol": "SecureCookieSessionInterface.open_session",
|
||||
"start_line": 338,
|
||||
"end_line": 349,
|
||||
"sha256": "76ebd81a608687f1f772032ecb6a1e4a3ac2b02bcbca517808128e5e9374426b"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "click-01",
|
||||
"repository": "click",
|
||||
"query": "Find where a supplied option value is normalized and matched against the allowed choices.",
|
||||
"targets": [
|
||||
{
|
||||
"path": "src/click/types.py",
|
||||
"symbol": "Choice.convert",
|
||||
"start_line": 344,
|
||||
"end_line": 358,
|
||||
"sha256": "2814d1c4ddbc711d5567999bf62257f7c3104b0fe9f521b3963a9f108f33e53c"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "click-02",
|
||||
"repository": "click",
|
||||
"query": "Locate where a filesystem argument is checked for existence, readability, writability, and allowed file or directory type.",
|
||||
"targets": [
|
||||
{
|
||||
"path": "src/click/types.py",
|
||||
"symbol": "Path.convert",
|
||||
"start_line": 930,
|
||||
"end_line": 995,
|
||||
"sha256": "2814d1c4ddbc711d5567999bf62257f7c3104b0fe9f521b3963a9f108f33e53c"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "click-03",
|
||||
"repository": "click",
|
||||
"query": "Find where a date argument is parsed by trying several accepted formats and reports a failure if none match.",
|
||||
"targets": [
|
||||
{
|
||||
"path": "src/click/types.py",
|
||||
"symbol": "DateTime.convert",
|
||||
"start_line": 448,
|
||||
"end_line": 466,
|
||||
"sha256": "2814d1c4ddbc711d5567999bf62257f7c3104b0fe9f521b3963a9f108f33e53c"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "click-04",
|
||||
"repository": "click",
|
||||
"query": "Where is an option value from an environment variable split and grouped for multiple arguments?",
|
||||
"targets": [
|
||||
{
|
||||
"path": "src/click/core.py",
|
||||
"symbol": "Option.value_from_envvar",
|
||||
"start_line": 2983,
|
||||
"end_line": 2996,
|
||||
"sha256": "814869352f5c14119d1175dd8ac1951be791bc67f8f514f215a831ba5ab0745c"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "click-05",
|
||||
"repository": "click",
|
||||
"query": "Locate where a command group resolves a subcommand name, retries normalized names, and rejects unknown commands.",
|
||||
"targets": [
|
||||
{
|
||||
"path": "src/click/core.py",
|
||||
"symbol": "Group.resolve_command",
|
||||
"start_line": 1867,
|
||||
"end_line": 1889,
|
||||
"sha256": "814869352f5c14119d1175dd8ac1951be791bc67f8f514f215a831ba5ab0745c"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "click-06",
|
||||
"repository": "click",
|
||||
"query": "Find the helper that enters a context manager and registers its cleanup with the command context.",
|
||||
"targets": [
|
||||
{
|
||||
"path": "src/click/core.py",
|
||||
"symbol": "Context.with_resource",
|
||||
"start_line": 598,
|
||||
"end_line": 598,
|
||||
"sha256": "814869352f5c14119d1175dd8ac1951be791bc67f8f514f215a831ba5ab0745c"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "click-07",
|
||||
"repository": "click",
|
||||
"query": "Find where each element of a tuple argument is converted using its corresponding parameter type.",
|
||||
"targets": [
|
||||
{
|
||||
"path": "src/click/types.py",
|
||||
"symbol": "Tuple.convert",
|
||||
"start_line": 1049,
|
||||
"end_line": 1065,
|
||||
"sha256": "2814d1c4ddbc711d5567999bf62257f7c3104b0fe9f521b3963a9f108f33e53c"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "click-08",
|
||||
"repository": "click",
|
||||
"query": "Locate the interactive yes-or-no question loop that retries invalid answers and can abort on rejection.",
|
||||
"targets": [
|
||||
{
|
||||
"path": "src/click/termui.py",
|
||||
"symbol": "confirm",
|
||||
"start_line": 223,
|
||||
"end_line": 252,
|
||||
"sha256": "bc062b282d9aedffcd7c42210138445587dafff89be5741b4d2086b499400510"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "click-09",
|
||||
"repository": "click",
|
||||
"query": "Find the testing helper that temporarily changes the current directory and removes its temporary directory on exit.",
|
||||
"targets": [
|
||||
{
|
||||
"path": "src/click/testing.py",
|
||||
"symbol": "CliRunner.isolated_filesystem",
|
||||
"start_line": 552,
|
||||
"end_line": 565,
|
||||
"sha256": "d9e2dd01a0890864e83f94f0f9737c263ac36f7c1b2487c047183777472b5b93"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "click-10",
|
||||
"repository": "click",
|
||||
"query": "Where does a lazily opened file acquire its actual stream and translate operating-system errors into a file error?",
|
||||
"targets": [
|
||||
{
|
||||
"path": "src/click/utils.py",
|
||||
"symbol": "LazyFile.open",
|
||||
"start_line": 156,
|
||||
"end_line": 167,
|
||||
"sha256": "6f5326faeb040c11ed13070f96d3c89f7cd22b89f0d8a4e9e460bbfe849394ba"
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
},
|
||||
"suite_sha256": "d06effdf41ee2e38bf8f44629949d9b4491067986fc5a73aef22cf80c5a5bf96",
|
||||
"model": {
|
||||
"backend": "keyword",
|
||||
"model": null,
|
||||
"max_query_terms": 12,
|
||||
"window_radius_lines": 12,
|
||||
"max_read_calls": 3,
|
||||
"ranking": "sum of 1/log2(2+returned_term_matches) for distinct terms in each window",
|
||||
"search": "one separate grep per term, each with the standard bounded output",
|
||||
"stopwords": [
|
||||
"after",
|
||||
"and",
|
||||
"are",
|
||||
"before",
|
||||
"code",
|
||||
"containing",
|
||||
"current",
|
||||
"does",
|
||||
"find",
|
||||
"for",
|
||||
"from",
|
||||
"function",
|
||||
"given",
|
||||
"handles",
|
||||
"handling",
|
||||
"how",
|
||||
"implementation",
|
||||
"into",
|
||||
"locate",
|
||||
"method",
|
||||
"repository",
|
||||
"return",
|
||||
"returns",
|
||||
"source",
|
||||
"that",
|
||||
"the",
|
||||
"this",
|
||||
"used",
|
||||
"using",
|
||||
"what",
|
||||
"when",
|
||||
"where",
|
||||
"which",
|
||||
"with"
|
||||
]
|
||||
},
|
||||
"source_sha256": {
|
||||
"__init__.py": "dcd2b573883b8068e806e3052adf9c03728f1cf621d35ff013179de76f76c702",
|
||||
"text.py": "8db7b80ee446480175f7691872303c8b72fd014604ad4fb575b359cee71f1938",
|
||||
"io.py": "b29c21db767ea9773281b17fe0754375153d18fb64f314907528f00d584a047f",
|
||||
"data.py": "20a891eff11d555d2301b62f4953146603efc87b03da4802f136f694ebdba78d",
|
||||
"encoder.py": "cb5671c08f28a70ffd0e2a7033528efc3581896a27242d55eb250ce922ab038f",
|
||||
"metrics.py": "cb93d30878c0c1006f06d1d0c4cbb188f90c6f4fc65f69f9a3b11d847f28ad27",
|
||||
"train.py": "cf20f6189967cfc88f3849bc3a225f2680c42026d53f204d9d5afc05fe88a2d6",
|
||||
"lexical.py": "6c47c834c6bbf7b317e2514057eee7444720581f5ec452ecb01ac666397b86b3",
|
||||
"symbols.py": "a377d6aa2c6b479c2ee115dd8e4859f0c06d188ae97b86fa1c7133885ee32d86",
|
||||
"index.py": "e8b0a666b41fed48d98616d5dcdd104e59c4308ac990281b463e5ee68f97877f",
|
||||
"evaluate.py": "29c942b9cbfff21ba8514511e27752527f0ec5e2cad5fbf57ec3b220fc7acd0b",
|
||||
"__main__.py": "ab30d9b696e41ece67094e19c87b1a31fc0ebce8d3afa79dc8eaf561f8fb4895",
|
||||
"download.py": "24f9802fb7cc82b81f6b1cb93e9244352d7b86feeb0a54ec055fb3e8d0a6a4d7",
|
||||
"cli.py": "d081962fc459808c65bd4c1d389e12ad9a9466df0a1374c8095a8bd5ccbb369e",
|
||||
"scout.py": "c3150baae51f37646305fee1d2c61e7b37f2ec9c4a0ba7f92ab4d7d6b1710161",
|
||||
"server.py": "483fe8051012249e38a6836cbe217d8c581d033e4d19026a29a7bd5c83878190",
|
||||
"live_tools.py": "e10fb6aaeaa6d7c4a5a35c0c6a210be4a4c283b18d215767d6bee0bac707d14c",
|
||||
"local_policy.py": "b0711b46a6399dfbeeecdcc045fb7392030e187d7e21c6e4fa9b51e6fc8056f4",
|
||||
"agent.py": "47357a900e9696b7ec5268eb031067eb0069faaeef92fd3159bd822d6bc67cdb",
|
||||
"live_server.py": "ccfb3c63115fe991c2f5f22c05f794d89c90d783b52f8ddbb668a8541e3af6fe",
|
||||
"native_protocol.py": "3a4d16c1d2cd90cb7d81e635f40d721270b0d541f454557a6d14e80b6b1073a7",
|
||||
"eval_live.py": "38250d45fd7e60c23c4c86f5819242736f29df8af384f92713128c84ffd734ff",
|
||||
"prepare_live.py": "0675173c5e9c116c995c271dc06de914388f19eb83779c07e3009000a810290b",
|
||||
"live_data.py": "fd0bd5b118b2437146e08ef921dc4aa8a1da24e69bcbdd3efc5ff6e9c4fe8a78",
|
||||
"transformers_policy.py": "e8e8a81b70eb37563973e640534b425a7ea9fc7d398a39dc8d8f71b9be3f1b89",
|
||||
"train_policy.py": "785760ddb1373cc7b37b8e0bd4afa02560d8ebb512cf401f932bbca541f1e930",
|
||||
"keyword_baseline.py": "5fe18c76b4c6e085c358ab5c490d717ffdd82b917a2882cf826fa26c57f64816"
|
||||
},
|
||||
"max_rounds": null,
|
||||
"max_chars": 6000,
|
||||
"timeout_seconds": null,
|
||||
"context": 0,
|
||||
"max_generation_tokens": 0,
|
||||
"tokenizer": null,
|
||||
"temperature": 0,
|
||||
"seed": 42,
|
||||
"scope": "Single-function localization; no large-model baseline; public repositories may have appeared in the base model's pretraining."
|
||||
}
|
||||
@@ -0,0 +1,84 @@
|
||||
{
|
||||
"tasks": 30,
|
||||
"target_hit_rate": 0.06666666666666667,
|
||||
"file_hit_rate": 0.5,
|
||||
"macro_line_precision": 0.007655712050078247,
|
||||
"macro_line_recall": 0.06793650793650793,
|
||||
"macro_line_f1": 0.013448935956491168,
|
||||
"latency_median_seconds": 0.2294759239994164,
|
||||
"latency_p95_seconds": 0.2956961605472316,
|
||||
"statuses": {
|
||||
"completed": 30
|
||||
},
|
||||
"total_tool_errors": 0,
|
||||
"total_invalid_actions": 0,
|
||||
"mean_rounds": 0,
|
||||
"mean_tool_calls": 11.566666666666666,
|
||||
"total_input_tokens": 0,
|
||||
"total_output_tokens": 0,
|
||||
"mean_returned_chars": 2960.1666666666665,
|
||||
"per_repository": {
|
||||
"requests": {
|
||||
"tasks": 10,
|
||||
"target_hit_rate": 0.1,
|
||||
"file_hit_rate": 0.6,
|
||||
"macro_line_precision": 0.013333333333333332,
|
||||
"macro_line_recall": 0.08333333333333334,
|
||||
"macro_line_f1": 0.02298850574712644,
|
||||
"latency_median_seconds": 0.23232812149944948,
|
||||
"latency_p95_seconds": 0.2572449309998774,
|
||||
"statuses": {
|
||||
"completed": 10
|
||||
},
|
||||
"total_tool_errors": 0,
|
||||
"total_invalid_actions": 0,
|
||||
"mean_rounds": 0,
|
||||
"mean_tool_calls": 11.5,
|
||||
"total_input_tokens": 0,
|
||||
"total_output_tokens": 0,
|
||||
"mean_returned_chars": 2925.2
|
||||
},
|
||||
"flask": {
|
||||
"tasks": 10,
|
||||
"target_hit_rate": 0.1,
|
||||
"file_hit_rate": 0.3,
|
||||
"macro_line_precision": 0.005633802816901409,
|
||||
"macro_line_recall": 0.1,
|
||||
"macro_line_f1": 0.010666666666666668,
|
||||
"latency_median_seconds": 0.23652058899824624,
|
||||
"latency_p95_seconds": 0.2910236757990788,
|
||||
"statuses": {
|
||||
"completed": 10
|
||||
},
|
||||
"total_tool_errors": 0,
|
||||
"total_invalid_actions": 0,
|
||||
"mean_rounds": 0,
|
||||
"mean_tool_calls": 11.2,
|
||||
"total_input_tokens": 0,
|
||||
"total_output_tokens": 0,
|
||||
"mean_returned_chars": 3031.8
|
||||
},
|
||||
"click": {
|
||||
"tasks": 10,
|
||||
"target_hit_rate": 0,
|
||||
"file_hit_rate": 0.6,
|
||||
"macro_line_precision": 0.004,
|
||||
"macro_line_recall": 0.020476190476190474,
|
||||
"macro_line_f1": 0.006691635455680399,
|
||||
"latency_median_seconds": 0.21134028099913849,
|
||||
"latency_p95_seconds": 0.2956961605472316,
|
||||
"statuses": {
|
||||
"completed": 10
|
||||
},
|
||||
"total_tool_errors": 0,
|
||||
"total_invalid_actions": 0,
|
||||
"mean_rounds": 0,
|
||||
"mean_tool_calls": 12,
|
||||
"total_input_tokens": 0,
|
||||
"total_output_tokens": 0,
|
||||
"mean_returned_chars": 2923.5
|
||||
}
|
||||
},
|
||||
"sampled_peak_gpu_memory_mib": null,
|
||||
"gpu_sampling_interval_seconds": 1
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,34 @@
|
||||
{
|
||||
"initialization_seconds": 22.52389923999999,
|
||||
"tools": [
|
||||
"scout_live_search"
|
||||
],
|
||||
"same_resident_server_calls": 2,
|
||||
"results": [
|
||||
{
|
||||
"status": "budget_exhausted",
|
||||
"elapsed_seconds": 31.885786627000016,
|
||||
"rounds": 6,
|
||||
"tool_calls": 5,
|
||||
"tool_errors": 1,
|
||||
"invalid_actions": 2,
|
||||
"input_tokens": 17537,
|
||||
"output_tokens": 168,
|
||||
"returned_chars": 0,
|
||||
"references": []
|
||||
},
|
||||
{
|
||||
"status": "budget_exhausted",
|
||||
"elapsed_seconds": 30.79203933000008,
|
||||
"rounds": 6,
|
||||
"tool_calls": 5,
|
||||
"tool_errors": 1,
|
||||
"invalid_actions": 2,
|
||||
"input_tokens": 17537,
|
||||
"output_tokens": 168,
|
||||
"returned_chars": 0,
|
||||
"references": []
|
||||
}
|
||||
],
|
||||
"scope": "Transport/resident inference check on an excluded development query; not task accuracy."
|
||||
}
|
||||
@@ -0,0 +1,524 @@
|
||||
{
|
||||
"suite": {
|
||||
"name": "live-search-v1",
|
||||
"scope": "30 hand-authored English single-function localization tasks in three public Python repositories. Development benchmark, not a downstream coding-task or contamination-free evaluation.",
|
||||
"repositories": {
|
||||
"requests": {
|
||||
"commit": "b25c87d7cb8d6a18a37fa12442b5f883f9e41741",
|
||||
"url": "https://github.com/psf/requests.git"
|
||||
},
|
||||
"flask": {
|
||||
"commit": "2c1b30d0503cfb064f1cb252e6614a06915a362a",
|
||||
"url": "https://github.com/pallets/flask.git"
|
||||
},
|
||||
"click": {
|
||||
"commit": "fd183b2ced1cb5857784fe7fb22f4982f671f098",
|
||||
"url": "https://github.com/pallets/click.git"
|
||||
}
|
||||
},
|
||||
"tasks": [
|
||||
{
|
||||
"id": "requests-01",
|
||||
"repository": "requests",
|
||||
"query": "Find where an unsuccessful HTTP status becomes an exception containing the server reason and URL.",
|
||||
"targets": [
|
||||
{
|
||||
"path": "src/requests/models.py",
|
||||
"symbol": "Response.raise_for_status",
|
||||
"start_line": 1002,
|
||||
"end_line": 1026,
|
||||
"sha256": "32365d67893bb67c3ed67cf93ca4a18e63e6ab29342fa0dc8b09c59e06ff564e"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "requests-02",
|
||||
"repository": "requests",
|
||||
"query": "Locate the response iterator that preserves an incomplete trailing line across downloaded chunks.",
|
||||
"targets": [
|
||||
{
|
||||
"path": "src/requests/models.py",
|
||||
"symbol": "Response.iter_lines",
|
||||
"start_line": 867,
|
||||
"end_line": 888,
|
||||
"sha256": "32365d67893bb67c3ed67cf93ca4a18e63e6ab29342fa0dc8b09c59e06ff564e"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "requests-03",
|
||||
"repository": "requests",
|
||||
"query": "Where is it decided whether credentials may survive a redirect to another hostname, port, or protocol?",
|
||||
"targets": [
|
||||
{
|
||||
"path": "src/requests/sessions.py",
|
||||
"symbol": "SessionRedirectMixin.should_strip_auth",
|
||||
"start_line": 129,
|
||||
"end_line": 157,
|
||||
"sha256": "0a5d5da449ce7f0af3ccf6e4bbe7a67a935e37846dff4ff9f08cb6c7e2464e6f"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "requests-04",
|
||||
"repository": "requests",
|
||||
"query": "Find the logic that changes the HTTP verb when following 301, 302, or 303 redirects.",
|
||||
"targets": [
|
||||
{
|
||||
"path": "src/requests/sessions.py",
|
||||
"symbol": "SessionRedirectMixin.rebuild_method",
|
||||
"start_line": 337,
|
||||
"end_line": 353,
|
||||
"sha256": "0a5d5da449ce7f0af3ccf6e4bbe7a67a935e37846dff4ff9f08cb6c7e2464e6f"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "requests-05",
|
||||
"repository": "requests",
|
||||
"query": "Locate where environment proxy settings and certificate bundle variables are merged with session options.",
|
||||
"targets": [
|
||||
{
|
||||
"path": "src/requests/sessions.py",
|
||||
"symbol": "Session.merge_environment_settings",
|
||||
"start_line": 757,
|
||||
"end_line": 779,
|
||||
"sha256": "0a5d5da449ce7f0af3ccf6e4bbe7a67a935e37846dff4ff9f08cb6c7e2464e6f"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "requests-06",
|
||||
"repository": "requests",
|
||||
"query": "Find where a 401 digest challenge causes the original request to be copied and sent again with authentication.",
|
||||
"targets": [
|
||||
{
|
||||
"path": "src/requests/auth.py",
|
||||
"symbol": "HTTPDigestAuth.handle_401",
|
||||
"start_line": 250,
|
||||
"end_line": 283,
|
||||
"sha256": "905ef9b6a9cb72d67d31ffe19bd4d9223e1c4169cde6ec51cfca16b31e70991d"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "requests-07",
|
||||
"repository": "requests",
|
||||
"query": "Locate the helper that fills a cookie container from a mapping while optionally preserving existing names.",
|
||||
"targets": [
|
||||
{
|
||||
"path": "src/requests/cookies.py",
|
||||
"symbol": "cookiejar_from_dict",
|
||||
"start_line": 530,
|
||||
"end_line": 539,
|
||||
"sha256": "6cd8be8aa123e0d3d9d34fa86feac7bf392f39bccdde5129830de0ea9692dd7c"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "requests-08",
|
||||
"repository": "requests",
|
||||
"query": "Find where credentials are loaded from the users netrc file, including NETRC and home-directory lookup.",
|
||||
"targets": [
|
||||
{
|
||||
"path": "src/requests/utils.py",
|
||||
"symbol": "get_netrc_auth",
|
||||
"start_line": 210,
|
||||
"end_line": 248,
|
||||
"sha256": "5aa53ceab677c2f842fad42359c8ed1ff1c4299c1607789609957a496e4311d4"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "requests-09",
|
||||
"repository": "requests",
|
||||
"query": "Find the implementation that seeks a request body back to its saved position and fails for an unrewindable stream.",
|
||||
"targets": [
|
||||
{
|
||||
"path": "src/requests/utils.py",
|
||||
"symbol": "rewind_body",
|
||||
"start_line": 1075,
|
||||
"end_line": 1086,
|
||||
"sha256": "5aa53ceab677c2f842fad42359c8ed1ff1c4299c1607789609957a496e4311d4"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "requests-10",
|
||||
"repository": "requests",
|
||||
"query": "Where is a request URL validated, its international hostname encoded, and its query parameters appended?",
|
||||
"targets": [
|
||||
{
|
||||
"path": "src/requests/models.py",
|
||||
"symbol": "PreparedRequest.prepare_url",
|
||||
"start_line": 416,
|
||||
"end_line": 481,
|
||||
"sha256": "32365d67893bb67c3ed67cf93ca4a18e63e6ab29342fa0dc8b09c59e06ff564e"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "flask-01",
|
||||
"repository": "flask",
|
||||
"query": "Find the code that invokes the view function selected by the matched URL rule.",
|
||||
"targets": [
|
||||
{
|
||||
"path": "src/flask/app.py",
|
||||
"symbol": "Flask.dispatch_request",
|
||||
"start_line": 889,
|
||||
"end_line": 902,
|
||||
"sha256": "5c6aa0151b0b8018732280761d27eda0c4c83378630e21faadd57b73783720a7"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "flask-02",
|
||||
"repository": "flask",
|
||||
"query": "Where are view return values such as tuples, dictionaries and strings converted into a response object?",
|
||||
"targets": [
|
||||
{
|
||||
"path": "src/flask/app.py",
|
||||
"symbol": "Flask.make_response",
|
||||
"start_line": 1186,
|
||||
"end_line": 1269,
|
||||
"sha256": "5c6aa0151b0b8018732280761d27eda0c4c83378630e21faadd57b73783720a7"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "flask-03",
|
||||
"repository": "flask",
|
||||
"query": "Find where coroutine view functions are adapted for synchronous request handling.",
|
||||
"targets": [
|
||||
{
|
||||
"path": "src/flask/app.py",
|
||||
"symbol": "Flask.ensure_sync",
|
||||
"start_line": 975,
|
||||
"end_line": 978,
|
||||
"sha256": "5c6aa0151b0b8018732280761d27eda0c4c83378630e21faadd57b73783720a7"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "flask-04",
|
||||
"repository": "flask",
|
||||
"query": "Locate the processing that runs after-request callbacks and saves the session before returning the response.",
|
||||
"targets": [
|
||||
{
|
||||
"path": "src/flask/app.py",
|
||||
"symbol": "Flask.process_response",
|
||||
"start_line": 1311,
|
||||
"end_line": 1324,
|
||||
"sha256": "5c6aa0151b0b8018732280761d27eda0c4c83378630e21faadd57b73783720a7"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "flask-05",
|
||||
"repository": "flask",
|
||||
"query": "Find where before-request handlers can short-circuit normal request dispatch by returning a value.",
|
||||
"targets": [
|
||||
{
|
||||
"path": "src/flask/app.py",
|
||||
"symbol": "Flask.preprocess_request",
|
||||
"start_line": 1281,
|
||||
"end_line": 1296,
|
||||
"sha256": "5c6aa0151b0b8018732280761d27eda0c4c83378630e21faadd57b73783720a7"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "flask-06",
|
||||
"repository": "flask",
|
||||
"query": "Where is a streaming generator wrapped so the request context remains active while producing its items?",
|
||||
"targets": [
|
||||
{
|
||||
"path": "src/flask/helpers.py",
|
||||
"symbol": "stream_with_context",
|
||||
"start_line": 104,
|
||||
"end_line": 143,
|
||||
"sha256": "ac96607bbfc9dbcf09d5442fe7e90d7f8a046b0df7d8f3fc3535b44112016d71"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "flask-07",
|
||||
"repository": "flask",
|
||||
"query": "Find where a categorized one-time message is stored in the session and a notification signal is emitted.",
|
||||
"targets": [
|
||||
{
|
||||
"path": "src/flask/helpers.py",
|
||||
"symbol": "flash",
|
||||
"start_line": 340,
|
||||
"end_line": 349,
|
||||
"sha256": "ac96607bbfc9dbcf09d5442fe7e90d7f8a046b0df7d8f3fc3535b44112016d71"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "flask-08",
|
||||
"repository": "flask",
|
||||
"query": "Locate where stored one-time messages are popped from the session, cached for the request, and filtered by category.",
|
||||
"targets": [
|
||||
{
|
||||
"path": "src/flask/helpers.py",
|
||||
"symbol": "get_flashed_messages",
|
||||
"start_line": 383,
|
||||
"end_line": 391,
|
||||
"sha256": "ac96607bbfc9dbcf09d5442fe7e90d7f8a046b0df7d8f3fc3535b44112016d71"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "flask-09",
|
||||
"repository": "flask",
|
||||
"query": "Find where the command-line startup reads dotenv files without replacing environment variables already set.",
|
||||
"targets": [
|
||||
{
|
||||
"path": "src/flask/cli.py",
|
||||
"symbol": "load_dotenv",
|
||||
"start_line": 740,
|
||||
"end_line": 771,
|
||||
"sha256": "3df87bdbe07196fa07d101c20ce7351ef5c6ecaab95deb5fdf3ccdcf690d4879"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "flask-10",
|
||||
"repository": "flask",
|
||||
"query": "Locate where a signed session cookie is verified and an invalid signature produces an empty session.",
|
||||
"targets": [
|
||||
{
|
||||
"path": "src/flask/sessions.py",
|
||||
"symbol": "SecureCookieSessionInterface.open_session",
|
||||
"start_line": 338,
|
||||
"end_line": 349,
|
||||
"sha256": "76ebd81a608687f1f772032ecb6a1e4a3ac2b02bcbca517808128e5e9374426b"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "click-01",
|
||||
"repository": "click",
|
||||
"query": "Find where a supplied option value is normalized and matched against the allowed choices.",
|
||||
"targets": [
|
||||
{
|
||||
"path": "src/click/types.py",
|
||||
"symbol": "Choice.convert",
|
||||
"start_line": 344,
|
||||
"end_line": 358,
|
||||
"sha256": "2814d1c4ddbc711d5567999bf62257f7c3104b0fe9f521b3963a9f108f33e53c"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "click-02",
|
||||
"repository": "click",
|
||||
"query": "Locate where a filesystem argument is checked for existence, readability, writability, and allowed file or directory type.",
|
||||
"targets": [
|
||||
{
|
||||
"path": "src/click/types.py",
|
||||
"symbol": "Path.convert",
|
||||
"start_line": 930,
|
||||
"end_line": 995,
|
||||
"sha256": "2814d1c4ddbc711d5567999bf62257f7c3104b0fe9f521b3963a9f108f33e53c"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "click-03",
|
||||
"repository": "click",
|
||||
"query": "Find where a date argument is parsed by trying several accepted formats and reports a failure if none match.",
|
||||
"targets": [
|
||||
{
|
||||
"path": "src/click/types.py",
|
||||
"symbol": "DateTime.convert",
|
||||
"start_line": 448,
|
||||
"end_line": 466,
|
||||
"sha256": "2814d1c4ddbc711d5567999bf62257f7c3104b0fe9f521b3963a9f108f33e53c"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "click-04",
|
||||
"repository": "click",
|
||||
"query": "Where is an option value from an environment variable split and grouped for multiple arguments?",
|
||||
"targets": [
|
||||
{
|
||||
"path": "src/click/core.py",
|
||||
"symbol": "Option.value_from_envvar",
|
||||
"start_line": 2983,
|
||||
"end_line": 2996,
|
||||
"sha256": "814869352f5c14119d1175dd8ac1951be791bc67f8f514f215a831ba5ab0745c"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "click-05",
|
||||
"repository": "click",
|
||||
"query": "Locate where a command group resolves a subcommand name, retries normalized names, and rejects unknown commands.",
|
||||
"targets": [
|
||||
{
|
||||
"path": "src/click/core.py",
|
||||
"symbol": "Group.resolve_command",
|
||||
"start_line": 1867,
|
||||
"end_line": 1889,
|
||||
"sha256": "814869352f5c14119d1175dd8ac1951be791bc67f8f514f215a831ba5ab0745c"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "click-06",
|
||||
"repository": "click",
|
||||
"query": "Find the helper that enters a context manager and registers its cleanup with the command context.",
|
||||
"targets": [
|
||||
{
|
||||
"path": "src/click/core.py",
|
||||
"symbol": "Context.with_resource",
|
||||
"start_line": 598,
|
||||
"end_line": 598,
|
||||
"sha256": "814869352f5c14119d1175dd8ac1951be791bc67f8f514f215a831ba5ab0745c"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "click-07",
|
||||
"repository": "click",
|
||||
"query": "Find where each element of a tuple argument is converted using its corresponding parameter type.",
|
||||
"targets": [
|
||||
{
|
||||
"path": "src/click/types.py",
|
||||
"symbol": "Tuple.convert",
|
||||
"start_line": 1049,
|
||||
"end_line": 1065,
|
||||
"sha256": "2814d1c4ddbc711d5567999bf62257f7c3104b0fe9f521b3963a9f108f33e53c"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "click-08",
|
||||
"repository": "click",
|
||||
"query": "Locate the interactive yes-or-no question loop that retries invalid answers and can abort on rejection.",
|
||||
"targets": [
|
||||
{
|
||||
"path": "src/click/termui.py",
|
||||
"symbol": "confirm",
|
||||
"start_line": 223,
|
||||
"end_line": 252,
|
||||
"sha256": "bc062b282d9aedffcd7c42210138445587dafff89be5741b4d2086b499400510"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "click-09",
|
||||
"repository": "click",
|
||||
"query": "Find the testing helper that temporarily changes the current directory and removes its temporary directory on exit.",
|
||||
"targets": [
|
||||
{
|
||||
"path": "src/click/testing.py",
|
||||
"symbol": "CliRunner.isolated_filesystem",
|
||||
"start_line": 552,
|
||||
"end_line": 565,
|
||||
"sha256": "d9e2dd01a0890864e83f94f0f9737c263ac36f7c1b2487c047183777472b5b93"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "click-10",
|
||||
"repository": "click",
|
||||
"query": "Where does a lazily opened file acquire its actual stream and translate operating-system errors into a file error?",
|
||||
"targets": [
|
||||
{
|
||||
"path": "src/click/utils.py",
|
||||
"symbol": "LazyFile.open",
|
||||
"start_line": 156,
|
||||
"end_line": 167,
|
||||
"sha256": "6f5326faeb040c11ed13070f96d3c89f7cd22b89f0d8a4e9e460bbfe849394ba"
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
},
|
||||
"suite_sha256": "d06effdf41ee2e38bf8f44629949d9b4491067986fc5a73aef22cf80c5a5bf96",
|
||||
"model": {
|
||||
"model": "openbmb/minicpm5:q4_K_M",
|
||||
"details": {
|
||||
"parent_model": "/Users/tianchi/.ollama/models/blobs/sha256-81b64d05a23b17b34c475f42b3e72fbde62d4b92cc34541f7a8031d0752deafa",
|
||||
"format": "gguf",
|
||||
"family": "llama",
|
||||
"families": [
|
||||
"llama"
|
||||
],
|
||||
"parameter_size": "1.1B",
|
||||
"quantization_level": "Q4_K_M"
|
||||
},
|
||||
"model_info": {
|
||||
"general.architecture": "llama",
|
||||
"general.basename": "MiniCPM5",
|
||||
"general.file_type": 15,
|
||||
"general.organization": "Openbmb",
|
||||
"general.parameter_count": 1080632832,
|
||||
"general.quantization_version": 2,
|
||||
"general.size_label": "1B",
|
||||
"general.type": "model",
|
||||
"llama.attention.head_count": 16,
|
||||
"llama.attention.head_count_kv": 2,
|
||||
"llama.attention.key_length": 128,
|
||||
"llama.attention.layer_norm_rms_epsilon": 1e-06,
|
||||
"llama.attention.value_length": 128,
|
||||
"llama.block_count": 24,
|
||||
"llama.context_length": 131072,
|
||||
"llama.embedding_length": 1536,
|
||||
"llama.feed_forward_length": 4608,
|
||||
"llama.rope.dimension_count": 128,
|
||||
"llama.rope.freq_base": 5000000,
|
||||
"llama.vocab_size": 130560,
|
||||
"tokenizer.ggml.add_bos_token": false,
|
||||
"tokenizer.ggml.add_eos_token": false,
|
||||
"tokenizer.ggml.add_sep_token": false,
|
||||
"tokenizer.ggml.add_space_prefix": false,
|
||||
"tokenizer.ggml.bos_token_id": 0,
|
||||
"tokenizer.ggml.eos_token_id": 1,
|
||||
"tokenizer.ggml.merges": null,
|
||||
"tokenizer.ggml.model": "gpt2",
|
||||
"tokenizer.ggml.padding_token_id": 1,
|
||||
"tokenizer.ggml.pre": "llama-bpe",
|
||||
"tokenizer.ggml.token_type": null,
|
||||
"tokenizer.ggml.tokens": null,
|
||||
"tokenizer.ggml.unknown_token_id": 130074
|
||||
}
|
||||
},
|
||||
"source_sha256": {
|
||||
"__init__.py": "dcd2b573883b8068e806e3052adf9c03728f1cf621d35ff013179de76f76c702",
|
||||
"text.py": "8db7b80ee446480175f7691872303c8b72fd014604ad4fb575b359cee71f1938",
|
||||
"io.py": "b29c21db767ea9773281b17fe0754375153d18fb64f314907528f00d584a047f",
|
||||
"data.py": "20a891eff11d555d2301b62f4953146603efc87b03da4802f136f694ebdba78d",
|
||||
"encoder.py": "cb5671c08f28a70ffd0e2a7033528efc3581896a27242d55eb250ce922ab038f",
|
||||
"metrics.py": "cb93d30878c0c1006f06d1d0c4cbb188f90c6f4fc65f69f9a3b11d847f28ad27",
|
||||
"train.py": "cf20f6189967cfc88f3849bc3a225f2680c42026d53f204d9d5afc05fe88a2d6",
|
||||
"lexical.py": "6c47c834c6bbf7b317e2514057eee7444720581f5ec452ecb01ac666397b86b3",
|
||||
"symbols.py": "a377d6aa2c6b479c2ee115dd8e4859f0c06d188ae97b86fa1c7133885ee32d86",
|
||||
"index.py": "e8b0a666b41fed48d98616d5dcdd104e59c4308ac990281b463e5ee68f97877f",
|
||||
"evaluate.py": "29c942b9cbfff21ba8514511e27752527f0ec5e2cad5fbf57ec3b220fc7acd0b",
|
||||
"__main__.py": "ab30d9b696e41ece67094e19c87b1a31fc0ebce8d3afa79dc8eaf561f8fb4895",
|
||||
"download.py": "24f9802fb7cc82b81f6b1cb93e9244352d7b86feeb0a54ec055fb3e8d0a6a4d7",
|
||||
"cli.py": "d481911029446e0efdc0b882e1520ffb40d3dfd1343403c73e86ff2e1dc45d03",
|
||||
"scout.py": "c3150baae51f37646305fee1d2c61e7b37f2ec9c4a0ba7f92ab4d7d6b1710161",
|
||||
"server.py": "483fe8051012249e38a6836cbe217d8c581d033e4d19026a29a7bd5c83878190",
|
||||
"live_tools.py": "e10fb6aaeaa6d7c4a5a35c0c6a210be4a4c283b18d215767d6bee0bac707d14c",
|
||||
"local_policy.py": "6d2b9dee7cf2688e5ff41b487151704839cfe2689f3c266905d3448573d985ad",
|
||||
"agent.py": "726ba5b78dba4e2b47d4e088789f3f767c92871b9c5fe44a3e4634722a5e516c",
|
||||
"live_server.py": "fba6f173a4b3736b1eef75ca02cd46e45999d15d0dfee3039539ffa5780e0cbb",
|
||||
"native_protocol.py": "3a4d16c1d2cd90cb7d81e635f40d721270b0d541f454557a6d14e80b6b1073a7",
|
||||
"eval_live.py": "9a28d3d113f0d100e982fdb053cf67c8bceb1c17ff05fad8435f5a2985924bdd"
|
||||
},
|
||||
"max_rounds": 6,
|
||||
"max_chars": 6000,
|
||||
"timeout_seconds": 90,
|
||||
"context": 8192,
|
||||
"max_generation_tokens": 512,
|
||||
"tokenizer": "pinned_hf",
|
||||
"temperature": 0,
|
||||
"seed": 42,
|
||||
"scope": "Single-function localization; no fine-tuning; no large-model baseline; public repositories may have appeared in the base model's pretraining."
|
||||
}
|
||||
@@ -0,0 +1,88 @@
|
||||
{
|
||||
"tasks": 30,
|
||||
"target_hit_rate": 0,
|
||||
"file_hit_rate": 0.06666666666666667,
|
||||
"macro_line_precision": 0.0,
|
||||
"macro_line_recall": 0.0,
|
||||
"macro_line_f1": 0.0,
|
||||
"latency_median_seconds": 4.832371773001796,
|
||||
"latency_p95_seconds": 15.41486109450052,
|
||||
"statuses": {
|
||||
"budget_exhausted": 17,
|
||||
"completed": 13
|
||||
},
|
||||
"total_tool_errors": 22,
|
||||
"total_invalid_actions": 37,
|
||||
"mean_rounds": 5.366666666666666,
|
||||
"mean_tool_calls": 4.7,
|
||||
"total_input_tokens": 376239,
|
||||
"total_output_tokens": 9038,
|
||||
"mean_returned_chars": 592.6,
|
||||
"per_repository": {
|
||||
"requests": {
|
||||
"tasks": 10,
|
||||
"target_hit_rate": 0,
|
||||
"file_hit_rate": 0.1,
|
||||
"macro_line_precision": 0.0,
|
||||
"macro_line_recall": 0.0,
|
||||
"macro_line_f1": 0.0,
|
||||
"latency_median_seconds": 4.516331323498889,
|
||||
"latency_p95_seconds": 7.343977602400628,
|
||||
"statuses": {
|
||||
"budget_exhausted": 6,
|
||||
"completed": 4
|
||||
},
|
||||
"total_tool_errors": 12,
|
||||
"total_invalid_actions": 11,
|
||||
"mean_rounds": 5.5,
|
||||
"mean_tool_calls": 5,
|
||||
"total_input_tokens": 117586,
|
||||
"total_output_tokens": 1738,
|
||||
"mean_returned_chars": 502.5
|
||||
},
|
||||
"flask": {
|
||||
"tasks": 10,
|
||||
"target_hit_rate": 0,
|
||||
"file_hit_rate": 0.1,
|
||||
"macro_line_precision": 0.0,
|
||||
"macro_line_recall": 0.0,
|
||||
"macro_line_f1": 0.0,
|
||||
"latency_median_seconds": 4.9962479714995425,
|
||||
"latency_p95_seconds": 16.429801569399206,
|
||||
"statuses": {
|
||||
"budget_exhausted": 6,
|
||||
"completed": 4
|
||||
},
|
||||
"total_tool_errors": 3,
|
||||
"total_invalid_actions": 18,
|
||||
"mean_rounds": 5.3,
|
||||
"mean_tool_calls": 4.1,
|
||||
"total_input_tokens": 134717,
|
||||
"total_output_tokens": 3817,
|
||||
"mean_returned_chars": 478.8
|
||||
},
|
||||
"click": {
|
||||
"tasks": 10,
|
||||
"target_hit_rate": 0,
|
||||
"file_hit_rate": 0,
|
||||
"macro_line_precision": 0.0,
|
||||
"macro_line_recall": 0.0,
|
||||
"macro_line_f1": 0.0,
|
||||
"latency_median_seconds": 4.9123027345012815,
|
||||
"latency_p95_seconds": 14.853568369802815,
|
||||
"statuses": {
|
||||
"budget_exhausted": 5,
|
||||
"completed": 5
|
||||
},
|
||||
"total_tool_errors": 7,
|
||||
"total_invalid_actions": 8,
|
||||
"mean_rounds": 5.3,
|
||||
"mean_tool_calls": 5,
|
||||
"total_input_tokens": 123936,
|
||||
"total_output_tokens": 3483,
|
||||
"mean_returned_chars": 796.5
|
||||
}
|
||||
},
|
||||
"sampled_peak_gpu_memory_mib": 1213,
|
||||
"gpu_sampling_interval_seconds": 1
|
||||
}
|
||||
@@ -0,0 +1,796 @@
|
||||
[
|
||||
{
|
||||
"id": "requests-01",
|
||||
"repository": "requests",
|
||||
"scores": {
|
||||
"file_hit": false,
|
||||
"target_hit": false,
|
||||
"line_precision": 0.0,
|
||||
"line_recall": 0.0,
|
||||
"line_f1": 0.0,
|
||||
"returned_lines": 0,
|
||||
"overlap_lines": 0
|
||||
},
|
||||
"status": "budget_exhausted",
|
||||
"elapsed_seconds": 9.196679419001157,
|
||||
"rounds": 6,
|
||||
"tool_calls": 3,
|
||||
"tool_errors": 0,
|
||||
"invalid_actions": 4,
|
||||
"input_tokens": 14900,
|
||||
"output_tokens": 191,
|
||||
"returned_chars": 0,
|
||||
"references": []
|
||||
},
|
||||
{
|
||||
"id": "requests-02",
|
||||
"repository": "requests",
|
||||
"scores": {
|
||||
"file_hit": false,
|
||||
"target_hit": false,
|
||||
"line_precision": 0.0,
|
||||
"line_recall": 0.0,
|
||||
"line_f1": 0.0,
|
||||
"returned_lines": 0,
|
||||
"overlap_lines": 0
|
||||
},
|
||||
"status": "budget_exhausted",
|
||||
"elapsed_seconds": 5.079564270999981,
|
||||
"rounds": 6,
|
||||
"tool_calls": 7,
|
||||
"tool_errors": 4,
|
||||
"invalid_actions": 0,
|
||||
"input_tokens": 11250,
|
||||
"output_tokens": 244,
|
||||
"returned_chars": 0,
|
||||
"references": []
|
||||
},
|
||||
{
|
||||
"id": "requests-03",
|
||||
"repository": "requests",
|
||||
"scores": {
|
||||
"file_hit": false,
|
||||
"target_hit": false,
|
||||
"line_precision": 0.0,
|
||||
"line_recall": 0.0,
|
||||
"line_f1": 0.0,
|
||||
"returned_lines": 0,
|
||||
"overlap_lines": 0
|
||||
},
|
||||
"status": "budget_exhausted",
|
||||
"elapsed_seconds": 4.466771458995936,
|
||||
"rounds": 6,
|
||||
"tool_calls": 6,
|
||||
"tool_errors": 3,
|
||||
"invalid_actions": 1,
|
||||
"input_tokens": 11000,
|
||||
"output_tokens": 187,
|
||||
"returned_chars": 0,
|
||||
"references": []
|
||||
},
|
||||
{
|
||||
"id": "requests-04",
|
||||
"repository": "requests",
|
||||
"scores": {
|
||||
"file_hit": false,
|
||||
"target_hit": false,
|
||||
"line_precision": 0.0,
|
||||
"line_recall": 0.0,
|
||||
"line_f1": 0.0,
|
||||
"returned_lines": 0,
|
||||
"overlap_lines": 0
|
||||
},
|
||||
"status": "budget_exhausted",
|
||||
"elapsed_seconds": 4.912948177996441,
|
||||
"rounds": 6,
|
||||
"tool_calls": 3,
|
||||
"tool_errors": 0,
|
||||
"invalid_actions": 4,
|
||||
"input_tokens": 15665,
|
||||
"output_tokens": 195,
|
||||
"returned_chars": 0,
|
||||
"references": []
|
||||
},
|
||||
{
|
||||
"id": "requests-05",
|
||||
"repository": "requests",
|
||||
"scores": {
|
||||
"file_hit": false,
|
||||
"target_hit": false,
|
||||
"line_precision": 0.0,
|
||||
"line_recall": 0.0,
|
||||
"line_f1": 0.0,
|
||||
"returned_lines": 17,
|
||||
"overlap_lines": 0
|
||||
},
|
||||
"status": "completed",
|
||||
"elapsed_seconds": 3.2996576329969685,
|
||||
"rounds": 4,
|
||||
"tool_calls": 4,
|
||||
"tool_errors": 0,
|
||||
"invalid_actions": 0,
|
||||
"input_tokens": 7648,
|
||||
"output_tokens": 133,
|
||||
"returned_chars": 426,
|
||||
"references": [
|
||||
{
|
||||
"path": "src/requests/certs.py",
|
||||
"start_line": 1,
|
||||
"end_line": 17,
|
||||
"sha256": "67d49be35d009efea35054f2b2cd23145854eb1b2df1cb442ea7f2f04bf6de0c",
|
||||
"verified": true
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "requests-06",
|
||||
"repository": "requests",
|
||||
"scores": {
|
||||
"file_hit": false,
|
||||
"target_hit": false,
|
||||
"line_precision": 0.0,
|
||||
"line_recall": 0.0,
|
||||
"line_f1": 0.0,
|
||||
"returned_lines": 28,
|
||||
"overlap_lines": 0
|
||||
},
|
||||
"status": "completed",
|
||||
"elapsed_seconds": 4.778130480000982,
|
||||
"rounds": 6,
|
||||
"tool_calls": 6,
|
||||
"tool_errors": 2,
|
||||
"invalid_actions": 0,
|
||||
"input_tokens": 14241,
|
||||
"output_tokens": 188,
|
||||
"returned_chars": 574,
|
||||
"references": [
|
||||
{
|
||||
"path": "src/requests/utils.py",
|
||||
"start_line": 1,
|
||||
"end_line": 28,
|
||||
"sha256": "5aa53ceab677c2f842fad42359c8ed1ff1c4299c1607789609957a496e4311d4",
|
||||
"verified": true
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "requests-07",
|
||||
"repository": "requests",
|
||||
"scores": {
|
||||
"file_hit": true,
|
||||
"target_hit": false,
|
||||
"line_precision": 0.0,
|
||||
"line_recall": 0.0,
|
||||
"line_f1": 0.0,
|
||||
"returned_lines": 120,
|
||||
"overlap_lines": 0
|
||||
},
|
||||
"status": "completed",
|
||||
"elapsed_seconds": 3.3701258940054686,
|
||||
"rounds": 4,
|
||||
"tool_calls": 4,
|
||||
"tool_errors": 0,
|
||||
"invalid_actions": 0,
|
||||
"input_tokens": 8282,
|
||||
"output_tokens": 131,
|
||||
"returned_chars": 3419,
|
||||
"references": [
|
||||
{
|
||||
"path": "src/requests/cookies.py",
|
||||
"start_line": 1,
|
||||
"end_line": 120,
|
||||
"sha256": "6cd8be8aa123e0d3d9d34fa86feac7bf392f39bccdde5129830de0ea9692dd7c",
|
||||
"verified": true
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "requests-08",
|
||||
"repository": "requests",
|
||||
"scores": {
|
||||
"file_hit": false,
|
||||
"target_hit": false,
|
||||
"line_precision": 0.0,
|
||||
"line_recall": 0.0,
|
||||
"line_f1": 0.0,
|
||||
"returned_lines": 0,
|
||||
"overlap_lines": 0
|
||||
},
|
||||
"status": "budget_exhausted",
|
||||
"elapsed_seconds": 4.330414477000886,
|
||||
"rounds": 6,
|
||||
"tool_calls": 7,
|
||||
"tool_errors": 2,
|
||||
"invalid_actions": 0,
|
||||
"input_tokens": 11005,
|
||||
"output_tokens": 162,
|
||||
"returned_chars": 0,
|
||||
"references": []
|
||||
},
|
||||
{
|
||||
"id": "requests-09",
|
||||
"repository": "requests",
|
||||
"scores": {
|
||||
"file_hit": false,
|
||||
"target_hit": false,
|
||||
"line_precision": 0.0,
|
||||
"line_recall": 0.0,
|
||||
"line_f1": 0.0,
|
||||
"returned_lines": 0,
|
||||
"overlap_lines": 0
|
||||
},
|
||||
"status": "budget_exhausted",
|
||||
"elapsed_seconds": 4.565891188001842,
|
||||
"rounds": 6,
|
||||
"tool_calls": 6,
|
||||
"tool_errors": 1,
|
||||
"invalid_actions": 1,
|
||||
"input_tokens": 10899,
|
||||
"output_tokens": 154,
|
||||
"returned_chars": 0,
|
||||
"references": []
|
||||
},
|
||||
{
|
||||
"id": "requests-10",
|
||||
"repository": "requests",
|
||||
"scores": {
|
||||
"file_hit": false,
|
||||
"target_hit": false,
|
||||
"line_precision": 0.0,
|
||||
"line_recall": 0.0,
|
||||
"line_f1": 0.0,
|
||||
"returned_lines": 19,
|
||||
"overlap_lines": 0
|
||||
},
|
||||
"status": "completed",
|
||||
"elapsed_seconds": 4.182433162997768,
|
||||
"rounds": 5,
|
||||
"tool_calls": 4,
|
||||
"tool_errors": 0,
|
||||
"invalid_actions": 1,
|
||||
"input_tokens": 12696,
|
||||
"output_tokens": 153,
|
||||
"returned_chars": 606,
|
||||
"references": [
|
||||
{
|
||||
"path": "src/requests/api.py",
|
||||
"start_line": 2,
|
||||
"end_line": 20,
|
||||
"sha256": "fd96fd39aeedcd5222cd32b016b3e30c463d7a3b66fce9d2444467003c46b10b",
|
||||
"verified": true
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "flask-01",
|
||||
"repository": "flask",
|
||||
"scores": {
|
||||
"file_hit": false,
|
||||
"target_hit": false,
|
||||
"line_precision": 0.0,
|
||||
"line_recall": 0.0,
|
||||
"line_f1": 0.0,
|
||||
"returned_lines": 0,
|
||||
"overlap_lines": 0
|
||||
},
|
||||
"status": "budget_exhausted",
|
||||
"elapsed_seconds": 20.86598295099975,
|
||||
"rounds": 6,
|
||||
"tool_calls": 4,
|
||||
"tool_errors": 0,
|
||||
"invalid_actions": 3,
|
||||
"input_tokens": 19882,
|
||||
"output_tokens": 1603,
|
||||
"returned_chars": 0,
|
||||
"references": []
|
||||
},
|
||||
{
|
||||
"id": "flask-02",
|
||||
"repository": "flask",
|
||||
"scores": {
|
||||
"file_hit": false,
|
||||
"target_hit": false,
|
||||
"line_precision": 0.0,
|
||||
"line_recall": 0.0,
|
||||
"line_f1": 0.0,
|
||||
"returned_lines": 0,
|
||||
"overlap_lines": 0
|
||||
},
|
||||
"status": "budget_exhausted",
|
||||
"elapsed_seconds": 5.218971822003368,
|
||||
"rounds": 6,
|
||||
"tool_calls": 3,
|
||||
"tool_errors": 0,
|
||||
"invalid_actions": 4,
|
||||
"input_tokens": 15851,
|
||||
"output_tokens": 181,
|
||||
"returned_chars": 0,
|
||||
"references": []
|
||||
},
|
||||
{
|
||||
"id": "flask-03",
|
||||
"repository": "flask",
|
||||
"scores": {
|
||||
"file_hit": false,
|
||||
"target_hit": false,
|
||||
"line_precision": 0.0,
|
||||
"line_recall": 0.0,
|
||||
"line_f1": 0.0,
|
||||
"returned_lines": 0,
|
||||
"overlap_lines": 0
|
||||
},
|
||||
"status": "budget_exhausted",
|
||||
"elapsed_seconds": 5.0775881389999995,
|
||||
"rounds": 6,
|
||||
"tool_calls": 3,
|
||||
"tool_errors": 0,
|
||||
"invalid_actions": 4,
|
||||
"input_tokens": 15821,
|
||||
"output_tokens": 181,
|
||||
"returned_chars": 0,
|
||||
"references": []
|
||||
},
|
||||
{
|
||||
"id": "flask-04",
|
||||
"repository": "flask",
|
||||
"scores": {
|
||||
"file_hit": true,
|
||||
"target_hit": false,
|
||||
"line_precision": 0.0,
|
||||
"line_recall": 0.0,
|
||||
"line_f1": 0.0,
|
||||
"returned_lines": 30,
|
||||
"overlap_lines": 0
|
||||
},
|
||||
"status": "completed",
|
||||
"elapsed_seconds": 3.3676273439996294,
|
||||
"rounds": 4,
|
||||
"tool_calls": 4,
|
||||
"tool_errors": 0,
|
||||
"invalid_actions": 0,
|
||||
"input_tokens": 8506,
|
||||
"output_tokens": 111,
|
||||
"returned_chars": 1019,
|
||||
"references": [
|
||||
{
|
||||
"path": "src/flask/app.py",
|
||||
"start_line": 26,
|
||||
"end_line": 55,
|
||||
"sha256": "5c6aa0151b0b8018732280761d27eda0c4c83378630e21faadd57b73783720a7",
|
||||
"verified": true
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "flask-05",
|
||||
"repository": "flask",
|
||||
"scores": {
|
||||
"file_hit": false,
|
||||
"target_hit": false,
|
||||
"line_precision": 0.0,
|
||||
"line_recall": 0.0,
|
||||
"line_f1": 0.0,
|
||||
"returned_lines": 61,
|
||||
"overlap_lines": 0
|
||||
},
|
||||
"status": "completed",
|
||||
"elapsed_seconds": 3.3826584030030062,
|
||||
"rounds": 4,
|
||||
"tool_calls": 4,
|
||||
"tool_errors": 0,
|
||||
"invalid_actions": 0,
|
||||
"input_tokens": 8260,
|
||||
"output_tokens": 116,
|
||||
"returned_chars": 2700,
|
||||
"references": [
|
||||
{
|
||||
"path": "src/flask/__init__.py",
|
||||
"start_line": 1,
|
||||
"end_line": 61,
|
||||
"sha256": "987bc937d4b0b65d510ed8c2a82218c889e22bf499fe5fe1a94ca73b382927da",
|
||||
"verified": true
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "flask-06",
|
||||
"repository": "flask",
|
||||
"scores": {
|
||||
"file_hit": false,
|
||||
"target_hit": false,
|
||||
"line_precision": 0.0,
|
||||
"line_recall": 0.0,
|
||||
"line_f1": 0.0,
|
||||
"returned_lines": 1,
|
||||
"overlap_lines": 0
|
||||
},
|
||||
"status": "completed",
|
||||
"elapsed_seconds": 3.1846724950009957,
|
||||
"rounds": 4,
|
||||
"tool_calls": 4,
|
||||
"tool_errors": 0,
|
||||
"invalid_actions": 0,
|
||||
"input_tokens": 7710,
|
||||
"output_tokens": 115,
|
||||
"returned_chars": 66,
|
||||
"references": [
|
||||
{
|
||||
"path": "src/flask/app.py",
|
||||
"start_line": 1175,
|
||||
"end_line": 1175,
|
||||
"sha256": "5c6aa0151b0b8018732280761d27eda0c4c83378630e21faadd57b73783720a7",
|
||||
"verified": true
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "flask-07",
|
||||
"repository": "flask",
|
||||
"scores": {
|
||||
"file_hit": false,
|
||||
"target_hit": false,
|
||||
"line_precision": 0.0,
|
||||
"line_recall": 0.0,
|
||||
"line_f1": 0.0,
|
||||
"returned_lines": 29,
|
||||
"overlap_lines": 0
|
||||
},
|
||||
"status": "completed",
|
||||
"elapsed_seconds": 4.172228956005711,
|
||||
"rounds": 5,
|
||||
"tool_calls": 5,
|
||||
"tool_errors": 0,
|
||||
"invalid_actions": 0,
|
||||
"input_tokens": 11127,
|
||||
"output_tokens": 143,
|
||||
"returned_chars": 1003,
|
||||
"references": [
|
||||
{
|
||||
"path": "src/flask/sessions.py",
|
||||
"start_line": 24,
|
||||
"end_line": 52,
|
||||
"sha256": "76ebd81a608687f1f772032ecb6a1e4a3ac2b02bcbca517808128e5e9374426b",
|
||||
"verified": true
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "flask-08",
|
||||
"repository": "flask",
|
||||
"scores": {
|
||||
"file_hit": false,
|
||||
"target_hit": false,
|
||||
"line_precision": 0.0,
|
||||
"line_recall": 0.0,
|
||||
"line_f1": 0.0,
|
||||
"returned_lines": 0,
|
||||
"overlap_lines": 0
|
||||
},
|
||||
"status": "budget_exhausted",
|
||||
"elapsed_seconds": 4.9149078039990854,
|
||||
"rounds": 6,
|
||||
"tool_calls": 3,
|
||||
"tool_errors": 0,
|
||||
"invalid_actions": 4,
|
||||
"input_tokens": 16124,
|
||||
"output_tokens": 181,
|
||||
"returned_chars": 0,
|
||||
"references": []
|
||||
},
|
||||
{
|
||||
"id": "flask-09",
|
||||
"repository": "flask",
|
||||
"scores": {
|
||||
"file_hit": false,
|
||||
"target_hit": false,
|
||||
"line_precision": 0.0,
|
||||
"line_recall": 0.0,
|
||||
"line_f1": 0.0,
|
||||
"returned_lines": 0,
|
||||
"overlap_lines": 0
|
||||
},
|
||||
"status": "budget_exhausted",
|
||||
"elapsed_seconds": 7.128173559998686,
|
||||
"rounds": 6,
|
||||
"tool_calls": 6,
|
||||
"tool_errors": 2,
|
||||
"invalid_actions": 1,
|
||||
"input_tokens": 14589,
|
||||
"output_tokens": 430,
|
||||
"returned_chars": 0,
|
||||
"references": []
|
||||
},
|
||||
{
|
||||
"id": "flask-10",
|
||||
"repository": "flask",
|
||||
"scores": {
|
||||
"file_hit": false,
|
||||
"target_hit": false,
|
||||
"line_precision": 0.0,
|
||||
"line_recall": 0.0,
|
||||
"line_f1": 0.0,
|
||||
"returned_lines": 0,
|
||||
"overlap_lines": 0
|
||||
},
|
||||
"status": "budget_exhausted",
|
||||
"elapsed_seconds": 11.00780210299854,
|
||||
"rounds": 6,
|
||||
"tool_calls": 5,
|
||||
"tool_errors": 1,
|
||||
"invalid_actions": 2,
|
||||
"input_tokens": 16847,
|
||||
"output_tokens": 756,
|
||||
"returned_chars": 0,
|
||||
"references": []
|
||||
},
|
||||
{
|
||||
"id": "click-01",
|
||||
"repository": "click",
|
||||
"scores": {
|
||||
"file_hit": false,
|
||||
"target_hit": false,
|
||||
"line_precision": 0.0,
|
||||
"line_recall": 0.0,
|
||||
"line_f1": 0.0,
|
||||
"returned_lines": 0,
|
||||
"overlap_lines": 0
|
||||
},
|
||||
"status": "budget_exhausted",
|
||||
"elapsed_seconds": 4.937992402999953,
|
||||
"rounds": 6,
|
||||
"tool_calls": 5,
|
||||
"tool_errors": 1,
|
||||
"invalid_actions": 2,
|
||||
"input_tokens": 13358,
|
||||
"output_tokens": 175,
|
||||
"returned_chars": 0,
|
||||
"references": []
|
||||
},
|
||||
{
|
||||
"id": "click-02",
|
||||
"repository": "click",
|
||||
"scores": {
|
||||
"file_hit": false,
|
||||
"target_hit": false,
|
||||
"line_precision": 0.0,
|
||||
"line_recall": 0.0,
|
||||
"line_f1": 0.0,
|
||||
"returned_lines": 0,
|
||||
"overlap_lines": 0
|
||||
},
|
||||
"status": "budget_exhausted",
|
||||
"elapsed_seconds": 4.979381118995661,
|
||||
"rounds": 6,
|
||||
"tool_calls": 6,
|
||||
"tool_errors": 1,
|
||||
"invalid_actions": 1,
|
||||
"input_tokens": 13377,
|
||||
"output_tokens": 181,
|
||||
"returned_chars": 0,
|
||||
"references": []
|
||||
},
|
||||
{
|
||||
"id": "click-03",
|
||||
"repository": "click",
|
||||
"scores": {
|
||||
"file_hit": false,
|
||||
"target_hit": false,
|
||||
"line_precision": 0.0,
|
||||
"line_recall": 0.0,
|
||||
"line_f1": 0.0,
|
||||
"returned_lines": 0,
|
||||
"overlap_lines": 0
|
||||
},
|
||||
"status": "budget_exhausted",
|
||||
"elapsed_seconds": 5.384169265002129,
|
||||
"rounds": 6,
|
||||
"tool_calls": 7,
|
||||
"tool_errors": 1,
|
||||
"invalid_actions": 0,
|
||||
"input_tokens": 15965,
|
||||
"output_tokens": 188,
|
||||
"returned_chars": 0,
|
||||
"references": []
|
||||
},
|
||||
{
|
||||
"id": "click-04",
|
||||
"repository": "click",
|
||||
"scores": {
|
||||
"file_hit": false,
|
||||
"target_hit": false,
|
||||
"line_precision": 0.0,
|
||||
"line_recall": 0.0,
|
||||
"line_f1": 0.0,
|
||||
"returned_lines": 0,
|
||||
"overlap_lines": 0
|
||||
},
|
||||
"status": "budget_exhausted",
|
||||
"elapsed_seconds": 9.760484937003639,
|
||||
"rounds": 6,
|
||||
"tool_calls": 6,
|
||||
"tool_errors": 3,
|
||||
"invalid_actions": 1,
|
||||
"input_tokens": 13646,
|
||||
"output_tokens": 658,
|
||||
"returned_chars": 0,
|
||||
"references": []
|
||||
},
|
||||
{
|
||||
"id": "click-05",
|
||||
"repository": "click",
|
||||
"scores": {
|
||||
"file_hit": false,
|
||||
"target_hit": false,
|
||||
"line_precision": 0.0,
|
||||
"line_recall": 0.0,
|
||||
"line_f1": 0.0,
|
||||
"returned_lines": 118,
|
||||
"overlap_lines": 0
|
||||
},
|
||||
"status": "completed",
|
||||
"elapsed_seconds": 4.572031217998301,
|
||||
"rounds": 5,
|
||||
"tool_calls": 5,
|
||||
"tool_errors": 0,
|
||||
"invalid_actions": 0,
|
||||
"input_tokens": 12974,
|
||||
"output_tokens": 161,
|
||||
"returned_chars": 3406,
|
||||
"references": [
|
||||
{
|
||||
"path": "src/click/_termui_impl.py",
|
||||
"start_line": 3,
|
||||
"end_line": 120,
|
||||
"sha256": "0125e12e2f4840873426cf4a4124bedfe48b65c3deb7756acac05ff5681b6e92",
|
||||
"verified": true
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "click-06",
|
||||
"repository": "click",
|
||||
"scores": {
|
||||
"file_hit": false,
|
||||
"target_hit": false,
|
||||
"line_precision": 0.0,
|
||||
"line_recall": 0.0,
|
||||
"line_f1": 0.0,
|
||||
"returned_lines": 16,
|
||||
"overlap_lines": 0
|
||||
},
|
||||
"status": "completed",
|
||||
"elapsed_seconds": 4.88661306600261,
|
||||
"rounds": 6,
|
||||
"tool_calls": 5,
|
||||
"tool_errors": 1,
|
||||
"invalid_actions": 1,
|
||||
"input_tokens": 14005,
|
||||
"output_tokens": 179,
|
||||
"returned_chars": 469,
|
||||
"references": [
|
||||
{
|
||||
"path": "src/click/formatting.py",
|
||||
"start_line": 256,
|
||||
"end_line": 271,
|
||||
"sha256": "061ab1e105dd290f56e162a49c8c23e4a3ca166b5db863ae1aad72c3f4c72d9f",
|
||||
"verified": true
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "click-07",
|
||||
"repository": "click",
|
||||
"scores": {
|
||||
"file_hit": false,
|
||||
"target_hit": false,
|
||||
"line_precision": 0.0,
|
||||
"line_recall": 0.0,
|
||||
"line_f1": 0.0,
|
||||
"returned_lines": 1,
|
||||
"overlap_lines": 0
|
||||
},
|
||||
"status": "completed",
|
||||
"elapsed_seconds": 3.422570918999554,
|
||||
"rounds": 4,
|
||||
"tool_calls": 4,
|
||||
"tool_errors": 0,
|
||||
"invalid_actions": 0,
|
||||
"input_tokens": 9480,
|
||||
"output_tokens": 112,
|
||||
"returned_chars": 33,
|
||||
"references": [
|
||||
{
|
||||
"path": "src/click/__init__.py",
|
||||
"start_line": 63,
|
||||
"end_line": 63,
|
||||
"sha256": "e98c92d5a7b29276742d8c1e5a8ccd672d00f6767fd75c2660886fddc6d0ad8a",
|
||||
"verified": true
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "click-08",
|
||||
"repository": "click",
|
||||
"scores": {
|
||||
"file_hit": false,
|
||||
"target_hit": false,
|
||||
"line_precision": 0.0,
|
||||
"line_recall": 0.0,
|
||||
"line_f1": 0.0,
|
||||
"returned_lines": 120,
|
||||
"overlap_lines": 0
|
||||
},
|
||||
"status": "completed",
|
||||
"elapsed_seconds": 3.3780149169979268,
|
||||
"rounds": 4,
|
||||
"tool_calls": 4,
|
||||
"tool_errors": 0,
|
||||
"invalid_actions": 0,
|
||||
"input_tokens": 8385,
|
||||
"output_tokens": 110,
|
||||
"returned_chars": 3507,
|
||||
"references": [
|
||||
{
|
||||
"path": "src/click/core.py",
|
||||
"start_line": 1,
|
||||
"end_line": 120,
|
||||
"sha256": "814869352f5c14119d1175dd8ac1951be791bc67f8f514f215a831ba5ab0745c",
|
||||
"verified": true
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "click-09",
|
||||
"repository": "click",
|
||||
"scores": {
|
||||
"file_hit": false,
|
||||
"target_hit": false,
|
||||
"line_precision": 0.0,
|
||||
"line_recall": 0.0,
|
||||
"line_f1": 0.0,
|
||||
"returned_lines": 0,
|
||||
"overlap_lines": 0
|
||||
},
|
||||
"status": "budget_exhausted",
|
||||
"elapsed_seconds": 19.02063663300214,
|
||||
"rounds": 6,
|
||||
"tool_calls": 4,
|
||||
"tool_errors": 0,
|
||||
"invalid_actions": 3,
|
||||
"input_tokens": 13511,
|
||||
"output_tokens": 1613,
|
||||
"returned_chars": 0,
|
||||
"references": []
|
||||
},
|
||||
{
|
||||
"id": "click-10",
|
||||
"repository": "click",
|
||||
"scores": {
|
||||
"file_hit": false,
|
||||
"target_hit": false,
|
||||
"line_precision": 0.0,
|
||||
"line_recall": 0.0,
|
||||
"line_f1": 0.0,
|
||||
"returned_lines": 11,
|
||||
"overlap_lines": 0
|
||||
},
|
||||
"status": "completed",
|
||||
"elapsed_seconds": 3.321955243998673,
|
||||
"rounds": 4,
|
||||
"tool_calls": 4,
|
||||
"tool_errors": 0,
|
||||
"invalid_actions": 0,
|
||||
"input_tokens": 9235,
|
||||
"output_tokens": 106,
|
||||
"returned_chars": 550,
|
||||
"references": [
|
||||
{
|
||||
"path": "src/click/core.py",
|
||||
"start_line": 820,
|
||||
"end_line": 830,
|
||||
"sha256": "814869352f5c14119d1175dd8ac1951be791bc67f8f514f215a831ba5ab0745c",
|
||||
"verified": true
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
@@ -0,0 +1,749 @@
|
||||
{
|
||||
"scope": "Diagnostic: target body covered by any successful read during search. Not final returned evidence; combined reads have a larger aggregate source budget.",
|
||||
"variants": {
|
||||
"ollama_q4": {
|
||||
"tasks": 30,
|
||||
"read_target_hits": 1,
|
||||
"per_task": [
|
||||
{
|
||||
"id": "requests-01",
|
||||
"read_target": false,
|
||||
"successful_reads": 0,
|
||||
"unique_read_lines": 0
|
||||
},
|
||||
{
|
||||
"id": "requests-02",
|
||||
"read_target": false,
|
||||
"successful_reads": 0,
|
||||
"unique_read_lines": 0
|
||||
},
|
||||
{
|
||||
"id": "requests-03",
|
||||
"read_target": false,
|
||||
"successful_reads": 0,
|
||||
"unique_read_lines": 0
|
||||
},
|
||||
{
|
||||
"id": "requests-04",
|
||||
"read_target": false,
|
||||
"successful_reads": 0,
|
||||
"unique_read_lines": 0
|
||||
},
|
||||
{
|
||||
"id": "requests-05",
|
||||
"read_target": false,
|
||||
"successful_reads": 1,
|
||||
"unique_read_lines": 17
|
||||
},
|
||||
{
|
||||
"id": "requests-06",
|
||||
"read_target": false,
|
||||
"successful_reads": 1,
|
||||
"unique_read_lines": 28
|
||||
},
|
||||
{
|
||||
"id": "requests-07",
|
||||
"read_target": false,
|
||||
"successful_reads": 1,
|
||||
"unique_read_lines": 120
|
||||
},
|
||||
{
|
||||
"id": "requests-08",
|
||||
"read_target": false,
|
||||
"successful_reads": 0,
|
||||
"unique_read_lines": 0
|
||||
},
|
||||
{
|
||||
"id": "requests-09",
|
||||
"read_target": false,
|
||||
"successful_reads": 0,
|
||||
"unique_read_lines": 0
|
||||
},
|
||||
{
|
||||
"id": "requests-10",
|
||||
"read_target": false,
|
||||
"successful_reads": 1,
|
||||
"unique_read_lines": 19
|
||||
},
|
||||
{
|
||||
"id": "flask-01",
|
||||
"read_target": false,
|
||||
"successful_reads": 0,
|
||||
"unique_read_lines": 0
|
||||
},
|
||||
{
|
||||
"id": "flask-02",
|
||||
"read_target": false,
|
||||
"successful_reads": 0,
|
||||
"unique_read_lines": 0
|
||||
},
|
||||
{
|
||||
"id": "flask-03",
|
||||
"read_target": false,
|
||||
"successful_reads": 0,
|
||||
"unique_read_lines": 0
|
||||
},
|
||||
{
|
||||
"id": "flask-04",
|
||||
"read_target": false,
|
||||
"successful_reads": 1,
|
||||
"unique_read_lines": 30
|
||||
},
|
||||
{
|
||||
"id": "flask-05",
|
||||
"read_target": false,
|
||||
"successful_reads": 1,
|
||||
"unique_read_lines": 61
|
||||
},
|
||||
{
|
||||
"id": "flask-06",
|
||||
"read_target": false,
|
||||
"successful_reads": 1,
|
||||
"unique_read_lines": 1
|
||||
},
|
||||
{
|
||||
"id": "flask-07",
|
||||
"read_target": false,
|
||||
"successful_reads": 1,
|
||||
"unique_read_lines": 29
|
||||
},
|
||||
{
|
||||
"id": "flask-08",
|
||||
"read_target": false,
|
||||
"successful_reads": 0,
|
||||
"unique_read_lines": 0
|
||||
},
|
||||
{
|
||||
"id": "flask-09",
|
||||
"read_target": false,
|
||||
"successful_reads": 1,
|
||||
"unique_read_lines": 8
|
||||
},
|
||||
{
|
||||
"id": "flask-10",
|
||||
"read_target": true,
|
||||
"successful_reads": 1,
|
||||
"unique_read_lines": 100
|
||||
},
|
||||
{
|
||||
"id": "click-01",
|
||||
"read_target": false,
|
||||
"successful_reads": 0,
|
||||
"unique_read_lines": 0
|
||||
},
|
||||
{
|
||||
"id": "click-02",
|
||||
"read_target": false,
|
||||
"successful_reads": 2,
|
||||
"unique_read_lines": 3
|
||||
},
|
||||
{
|
||||
"id": "click-03",
|
||||
"read_target": false,
|
||||
"successful_reads": 3,
|
||||
"unique_read_lines": 3
|
||||
},
|
||||
{
|
||||
"id": "click-04",
|
||||
"read_target": false,
|
||||
"successful_reads": 0,
|
||||
"unique_read_lines": 0
|
||||
},
|
||||
{
|
||||
"id": "click-05",
|
||||
"read_target": false,
|
||||
"successful_reads": 1,
|
||||
"unique_read_lines": 120
|
||||
},
|
||||
{
|
||||
"id": "click-06",
|
||||
"read_target": false,
|
||||
"successful_reads": 1,
|
||||
"unique_read_lines": 16
|
||||
},
|
||||
{
|
||||
"id": "click-07",
|
||||
"read_target": false,
|
||||
"successful_reads": 1,
|
||||
"unique_read_lines": 1
|
||||
},
|
||||
{
|
||||
"id": "click-08",
|
||||
"read_target": false,
|
||||
"successful_reads": 1,
|
||||
"unique_read_lines": 120
|
||||
},
|
||||
{
|
||||
"id": "click-09",
|
||||
"read_target": false,
|
||||
"successful_reads": 1,
|
||||
"unique_read_lines": 12
|
||||
},
|
||||
{
|
||||
"id": "click-10",
|
||||
"read_target": false,
|
||||
"successful_reads": 1,
|
||||
"unique_read_lines": 11
|
||||
}
|
||||
]
|
||||
},
|
||||
"transformers_nf4": {
|
||||
"tasks": 30,
|
||||
"read_target_hits": 0,
|
||||
"per_task": [
|
||||
{
|
||||
"id": "requests-01",
|
||||
"read_target": false,
|
||||
"successful_reads": 0,
|
||||
"unique_read_lines": 0
|
||||
},
|
||||
{
|
||||
"id": "requests-02",
|
||||
"read_target": false,
|
||||
"successful_reads": 0,
|
||||
"unique_read_lines": 0
|
||||
},
|
||||
{
|
||||
"id": "requests-03",
|
||||
"read_target": false,
|
||||
"successful_reads": 1,
|
||||
"unique_read_lines": 1
|
||||
},
|
||||
{
|
||||
"id": "requests-04",
|
||||
"read_target": false,
|
||||
"successful_reads": 0,
|
||||
"unique_read_lines": 0
|
||||
},
|
||||
{
|
||||
"id": "requests-05",
|
||||
"read_target": false,
|
||||
"successful_reads": 0,
|
||||
"unique_read_lines": 0
|
||||
},
|
||||
{
|
||||
"id": "requests-06",
|
||||
"read_target": false,
|
||||
"successful_reads": 1,
|
||||
"unique_read_lines": 65
|
||||
},
|
||||
{
|
||||
"id": "requests-07",
|
||||
"read_target": false,
|
||||
"successful_reads": 1,
|
||||
"unique_read_lines": 2
|
||||
},
|
||||
{
|
||||
"id": "requests-08",
|
||||
"read_target": false,
|
||||
"successful_reads": 0,
|
||||
"unique_read_lines": 0
|
||||
},
|
||||
{
|
||||
"id": "requests-09",
|
||||
"read_target": false,
|
||||
"successful_reads": 0,
|
||||
"unique_read_lines": 0
|
||||
},
|
||||
{
|
||||
"id": "requests-10",
|
||||
"read_target": false,
|
||||
"successful_reads": 3,
|
||||
"unique_read_lines": 23
|
||||
},
|
||||
{
|
||||
"id": "flask-01",
|
||||
"read_target": false,
|
||||
"successful_reads": 1,
|
||||
"unique_read_lines": 1
|
||||
},
|
||||
{
|
||||
"id": "flask-02",
|
||||
"read_target": false,
|
||||
"successful_reads": 0,
|
||||
"unique_read_lines": 0
|
||||
},
|
||||
{
|
||||
"id": "flask-03",
|
||||
"read_target": false,
|
||||
"successful_reads": 1,
|
||||
"unique_read_lines": 77
|
||||
},
|
||||
{
|
||||
"id": "flask-04",
|
||||
"read_target": false,
|
||||
"successful_reads": 0,
|
||||
"unique_read_lines": 0
|
||||
},
|
||||
{
|
||||
"id": "flask-05",
|
||||
"read_target": false,
|
||||
"successful_reads": 0,
|
||||
"unique_read_lines": 0
|
||||
},
|
||||
{
|
||||
"id": "flask-06",
|
||||
"read_target": false,
|
||||
"successful_reads": 1,
|
||||
"unique_read_lines": 1
|
||||
},
|
||||
{
|
||||
"id": "flask-07",
|
||||
"read_target": false,
|
||||
"successful_reads": 0,
|
||||
"unique_read_lines": 0
|
||||
},
|
||||
{
|
||||
"id": "flask-08",
|
||||
"read_target": false,
|
||||
"successful_reads": 0,
|
||||
"unique_read_lines": 0
|
||||
},
|
||||
{
|
||||
"id": "flask-09",
|
||||
"read_target": false,
|
||||
"successful_reads": 1,
|
||||
"unique_read_lines": 15
|
||||
},
|
||||
{
|
||||
"id": "flask-10",
|
||||
"read_target": false,
|
||||
"successful_reads": 0,
|
||||
"unique_read_lines": 0
|
||||
},
|
||||
{
|
||||
"id": "click-01",
|
||||
"read_target": false,
|
||||
"successful_reads": 0,
|
||||
"unique_read_lines": 0
|
||||
},
|
||||
{
|
||||
"id": "click-02",
|
||||
"read_target": false,
|
||||
"successful_reads": 0,
|
||||
"unique_read_lines": 0
|
||||
},
|
||||
{
|
||||
"id": "click-03",
|
||||
"read_target": false,
|
||||
"successful_reads": 1,
|
||||
"unique_read_lines": 3
|
||||
},
|
||||
{
|
||||
"id": "click-04",
|
||||
"read_target": false,
|
||||
"successful_reads": 0,
|
||||
"unique_read_lines": 0
|
||||
},
|
||||
{
|
||||
"id": "click-05",
|
||||
"read_target": false,
|
||||
"successful_reads": 4,
|
||||
"unique_read_lines": 4
|
||||
},
|
||||
{
|
||||
"id": "click-06",
|
||||
"read_target": false,
|
||||
"successful_reads": 1,
|
||||
"unique_read_lines": 1
|
||||
},
|
||||
{
|
||||
"id": "click-07",
|
||||
"read_target": false,
|
||||
"successful_reads": 1,
|
||||
"unique_read_lines": 6
|
||||
},
|
||||
{
|
||||
"id": "click-08",
|
||||
"read_target": false,
|
||||
"successful_reads": 1,
|
||||
"unique_read_lines": 4
|
||||
},
|
||||
{
|
||||
"id": "click-09",
|
||||
"read_target": false,
|
||||
"successful_reads": 1,
|
||||
"unique_read_lines": 1
|
||||
},
|
||||
{
|
||||
"id": "click-10",
|
||||
"read_target": false,
|
||||
"successful_reads": 0,
|
||||
"unique_read_lines": 0
|
||||
}
|
||||
]
|
||||
},
|
||||
"adapter_v1": {
|
||||
"tasks": 30,
|
||||
"read_target_hits": 0,
|
||||
"per_task": [
|
||||
{
|
||||
"id": "requests-01",
|
||||
"read_target": false,
|
||||
"successful_reads": 1,
|
||||
"unique_read_lines": 40
|
||||
},
|
||||
{
|
||||
"id": "requests-02",
|
||||
"read_target": false,
|
||||
"successful_reads": 1,
|
||||
"unique_read_lines": 35
|
||||
},
|
||||
{
|
||||
"id": "requests-03",
|
||||
"read_target": false,
|
||||
"successful_reads": 1,
|
||||
"unique_read_lines": 1
|
||||
},
|
||||
{
|
||||
"id": "requests-04",
|
||||
"read_target": false,
|
||||
"successful_reads": 1,
|
||||
"unique_read_lines": 84
|
||||
},
|
||||
{
|
||||
"id": "requests-05",
|
||||
"read_target": false,
|
||||
"successful_reads": 0,
|
||||
"unique_read_lines": 0
|
||||
},
|
||||
{
|
||||
"id": "requests-06",
|
||||
"read_target": false,
|
||||
"successful_reads": 1,
|
||||
"unique_read_lines": 66
|
||||
},
|
||||
{
|
||||
"id": "requests-07",
|
||||
"read_target": false,
|
||||
"successful_reads": 1,
|
||||
"unique_read_lines": 78
|
||||
},
|
||||
{
|
||||
"id": "requests-08",
|
||||
"read_target": false,
|
||||
"successful_reads": 1,
|
||||
"unique_read_lines": 1
|
||||
},
|
||||
{
|
||||
"id": "requests-09",
|
||||
"read_target": false,
|
||||
"successful_reads": 0,
|
||||
"unique_read_lines": 0
|
||||
},
|
||||
{
|
||||
"id": "requests-10",
|
||||
"read_target": false,
|
||||
"successful_reads": 1,
|
||||
"unique_read_lines": 2
|
||||
},
|
||||
{
|
||||
"id": "flask-01",
|
||||
"read_target": false,
|
||||
"successful_reads": 1,
|
||||
"unique_read_lines": 118
|
||||
},
|
||||
{
|
||||
"id": "flask-02",
|
||||
"read_target": false,
|
||||
"successful_reads": 1,
|
||||
"unique_read_lines": 23
|
||||
},
|
||||
{
|
||||
"id": "flask-03",
|
||||
"read_target": false,
|
||||
"successful_reads": 1,
|
||||
"unique_read_lines": 33
|
||||
},
|
||||
{
|
||||
"id": "flask-04",
|
||||
"read_target": false,
|
||||
"successful_reads": 1,
|
||||
"unique_read_lines": 37
|
||||
},
|
||||
{
|
||||
"id": "flask-05",
|
||||
"read_target": false,
|
||||
"successful_reads": 1,
|
||||
"unique_read_lines": 53
|
||||
},
|
||||
{
|
||||
"id": "flask-06",
|
||||
"read_target": false,
|
||||
"successful_reads": 1,
|
||||
"unique_read_lines": 17
|
||||
},
|
||||
{
|
||||
"id": "flask-07",
|
||||
"read_target": false,
|
||||
"successful_reads": 1,
|
||||
"unique_read_lines": 37
|
||||
},
|
||||
{
|
||||
"id": "flask-08",
|
||||
"read_target": false,
|
||||
"successful_reads": 1,
|
||||
"unique_read_lines": 31
|
||||
},
|
||||
{
|
||||
"id": "flask-09",
|
||||
"read_target": false,
|
||||
"successful_reads": 1,
|
||||
"unique_read_lines": 21
|
||||
},
|
||||
{
|
||||
"id": "flask-10",
|
||||
"read_target": false,
|
||||
"successful_reads": 0,
|
||||
"unique_read_lines": 0
|
||||
},
|
||||
{
|
||||
"id": "click-01",
|
||||
"read_target": false,
|
||||
"successful_reads": 1,
|
||||
"unique_read_lines": 111
|
||||
},
|
||||
{
|
||||
"id": "click-02",
|
||||
"read_target": false,
|
||||
"successful_reads": 1,
|
||||
"unique_read_lines": 89
|
||||
},
|
||||
{
|
||||
"id": "click-03",
|
||||
"read_target": false,
|
||||
"successful_reads": 1,
|
||||
"unique_read_lines": 21
|
||||
},
|
||||
{
|
||||
"id": "click-04",
|
||||
"read_target": false,
|
||||
"successful_reads": 1,
|
||||
"unique_read_lines": 3
|
||||
},
|
||||
{
|
||||
"id": "click-05",
|
||||
"read_target": false,
|
||||
"successful_reads": 1,
|
||||
"unique_read_lines": 75
|
||||
},
|
||||
{
|
||||
"id": "click-06",
|
||||
"read_target": false,
|
||||
"successful_reads": 1,
|
||||
"unique_read_lines": 87
|
||||
},
|
||||
{
|
||||
"id": "click-07",
|
||||
"read_target": false,
|
||||
"successful_reads": 1,
|
||||
"unique_read_lines": 56
|
||||
},
|
||||
{
|
||||
"id": "click-08",
|
||||
"read_target": false,
|
||||
"successful_reads": 1,
|
||||
"unique_read_lines": 14
|
||||
},
|
||||
{
|
||||
"id": "click-09",
|
||||
"read_target": false,
|
||||
"successful_reads": 1,
|
||||
"unique_read_lines": 21
|
||||
},
|
||||
{
|
||||
"id": "click-10",
|
||||
"read_target": false,
|
||||
"successful_reads": 1,
|
||||
"unique_read_lines": 48
|
||||
}
|
||||
]
|
||||
},
|
||||
"adapter_v2": {
|
||||
"tasks": 30,
|
||||
"read_target_hits": 1,
|
||||
"per_task": [
|
||||
{
|
||||
"id": "requests-01",
|
||||
"read_target": false,
|
||||
"successful_reads": 0,
|
||||
"unique_read_lines": 0
|
||||
},
|
||||
{
|
||||
"id": "requests-02",
|
||||
"read_target": false,
|
||||
"successful_reads": 1,
|
||||
"unique_read_lines": 81
|
||||
},
|
||||
{
|
||||
"id": "requests-03",
|
||||
"read_target": false,
|
||||
"successful_reads": 1,
|
||||
"unique_read_lines": 101
|
||||
},
|
||||
{
|
||||
"id": "requests-04",
|
||||
"read_target": false,
|
||||
"successful_reads": 0,
|
||||
"unique_read_lines": 0
|
||||
},
|
||||
{
|
||||
"id": "requests-05",
|
||||
"read_target": false,
|
||||
"successful_reads": 0,
|
||||
"unique_read_lines": 0
|
||||
},
|
||||
{
|
||||
"id": "requests-06",
|
||||
"read_target": true,
|
||||
"successful_reads": 1,
|
||||
"unique_read_lines": 90
|
||||
},
|
||||
{
|
||||
"id": "requests-07",
|
||||
"read_target": false,
|
||||
"successful_reads": 0,
|
||||
"unique_read_lines": 0
|
||||
},
|
||||
{
|
||||
"id": "requests-08",
|
||||
"read_target": false,
|
||||
"successful_reads": 1,
|
||||
"unique_read_lines": 101
|
||||
},
|
||||
{
|
||||
"id": "requests-09",
|
||||
"read_target": false,
|
||||
"successful_reads": 0,
|
||||
"unique_read_lines": 0
|
||||
},
|
||||
{
|
||||
"id": "requests-10",
|
||||
"read_target": false,
|
||||
"successful_reads": 0,
|
||||
"unique_read_lines": 0
|
||||
},
|
||||
{
|
||||
"id": "flask-01",
|
||||
"read_target": false,
|
||||
"successful_reads": 0,
|
||||
"unique_read_lines": 0
|
||||
},
|
||||
{
|
||||
"id": "flask-02",
|
||||
"read_target": false,
|
||||
"successful_reads": 0,
|
||||
"unique_read_lines": 0
|
||||
},
|
||||
{
|
||||
"id": "flask-03",
|
||||
"read_target": false,
|
||||
"successful_reads": 0,
|
||||
"unique_read_lines": 0
|
||||
},
|
||||
{
|
||||
"id": "flask-04",
|
||||
"read_target": false,
|
||||
"successful_reads": 0,
|
||||
"unique_read_lines": 0
|
||||
},
|
||||
{
|
||||
"id": "flask-05",
|
||||
"read_target": false,
|
||||
"successful_reads": 0,
|
||||
"unique_read_lines": 0
|
||||
},
|
||||
{
|
||||
"id": "flask-06",
|
||||
"read_target": false,
|
||||
"successful_reads": 0,
|
||||
"unique_read_lines": 0
|
||||
},
|
||||
{
|
||||
"id": "flask-07",
|
||||
"read_target": false,
|
||||
"successful_reads": 1,
|
||||
"unique_read_lines": 74
|
||||
},
|
||||
{
|
||||
"id": "flask-08",
|
||||
"read_target": false,
|
||||
"successful_reads": 1,
|
||||
"unique_read_lines": 101
|
||||
},
|
||||
{
|
||||
"id": "flask-09",
|
||||
"read_target": false,
|
||||
"successful_reads": 1,
|
||||
"unique_read_lines": 101
|
||||
},
|
||||
{
|
||||
"id": "flask-10",
|
||||
"read_target": false,
|
||||
"successful_reads": 0,
|
||||
"unique_read_lines": 0
|
||||
},
|
||||
{
|
||||
"id": "click-01",
|
||||
"read_target": false,
|
||||
"successful_reads": 0,
|
||||
"unique_read_lines": 0
|
||||
},
|
||||
{
|
||||
"id": "click-02",
|
||||
"read_target": false,
|
||||
"successful_reads": 0,
|
||||
"unique_read_lines": 0
|
||||
},
|
||||
{
|
||||
"id": "click-03",
|
||||
"read_target": false,
|
||||
"successful_reads": 0,
|
||||
"unique_read_lines": 0
|
||||
},
|
||||
{
|
||||
"id": "click-04",
|
||||
"read_target": false,
|
||||
"successful_reads": 0,
|
||||
"unique_read_lines": 0
|
||||
},
|
||||
{
|
||||
"id": "click-05",
|
||||
"read_target": false,
|
||||
"successful_reads": 0,
|
||||
"unique_read_lines": 0
|
||||
},
|
||||
{
|
||||
"id": "click-06",
|
||||
"read_target": false,
|
||||
"successful_reads": 1,
|
||||
"unique_read_lines": 118
|
||||
},
|
||||
{
|
||||
"id": "click-07",
|
||||
"read_target": false,
|
||||
"successful_reads": 0,
|
||||
"unique_read_lines": 0
|
||||
},
|
||||
{
|
||||
"id": "click-08",
|
||||
"read_target": false,
|
||||
"successful_reads": 0,
|
||||
"unique_read_lines": 0
|
||||
},
|
||||
{
|
||||
"id": "click-09",
|
||||
"read_target": false,
|
||||
"successful_reads": 0,
|
||||
"unique_read_lines": 0
|
||||
},
|
||||
{
|
||||
"id": "click-10",
|
||||
"read_target": false,
|
||||
"successful_reads": 0,
|
||||
"unique_read_lines": 0
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,105 @@
|
||||
{"step": 0, "validation_loss": 0.33653362238636386, "seconds": 30.465305138997792}
|
||||
{"step": 1, "epoch": 1, "loss": 0.3705778233706951, "seconds": 38.971022692996485, "peak_cuda_mib": 2576.94140625}
|
||||
{"step": 2, "epoch": 1, "loss": 0.28090649645309895, "seconds": 47.42822287399758, "peak_cuda_mib": 2735.1513671875}
|
||||
{"step": 3, "epoch": 1, "loss": 0.14360825764015317, "seconds": 57.45774060900294, "peak_cuda_mib": 2735.1513671875}
|
||||
{"step": 4, "epoch": 1, "loss": 0.16286213952116668, "seconds": 66.76757446400006, "peak_cuda_mib": 2735.7275390625}
|
||||
{"step": 5, "epoch": 1, "loss": 0.19456972985062748, "seconds": 75.38471084200137, "peak_cuda_mib": 2735.7275390625}
|
||||
{"step": 6, "epoch": 1, "loss": 0.11313682279433124, "seconds": 85.06348957100272, "peak_cuda_mib": 2743.55859375}
|
||||
{"step": 7, "epoch": 1, "loss": 0.055731827684212476, "seconds": 95.18198007599858, "peak_cuda_mib": 2743.55859375}
|
||||
{"step": 8, "epoch": 1, "loss": 0.10752122453413904, "seconds": 103.28240856799675, "peak_cuda_mib": 2743.55859375}
|
||||
{"step": 9, "epoch": 1, "loss": 0.06660429344628938, "seconds": 112.8437792629993, "peak_cuda_mib": 2743.55859375}
|
||||
{"step": 10, "epoch": 1, "loss": 0.12484436182421632, "seconds": 121.91538363199652, "peak_cuda_mib": 2743.55859375}
|
||||
{"step": 11, "epoch": 1, "loss": 0.21446877851121826, "seconds": 130.63474446899636, "peak_cuda_mib": 2743.55859375}
|
||||
{"step": 12, "epoch": 1, "loss": 0.054695741571777035, "seconds": 140.17853507900145, "peak_cuda_mib": 2743.55859375}
|
||||
{"step": 13, "epoch": 1, "loss": 0.10972705773019698, "seconds": 149.61897696599772, "peak_cuda_mib": 2743.55859375}
|
||||
{"step": 14, "epoch": 1, "loss": 0.06535851817170624, "seconds": 159.07375226300064, "peak_cuda_mib": 2743.55859375}
|
||||
{"step": 15, "epoch": 1, "loss": 0.02624401665161713, "seconds": 169.0047627890017, "peak_cuda_mib": 2743.55859375}
|
||||
{"step": 16, "epoch": 1, "loss": 0.08463715268590022, "seconds": 179.18186750099994, "peak_cuda_mib": 2769.76025390625}
|
||||
{"step": 17, "epoch": 1, "loss": 0.11441127636499004, "seconds": 188.12793054299982, "peak_cuda_mib": 2769.76025390625}
|
||||
{"step": 18, "epoch": 1, "loss": 0.10806689025594096, "seconds": 197.5903550840012, "peak_cuda_mib": 2769.76025390625}
|
||||
{"step": 19, "epoch": 1, "loss": 0.08541615212379838, "seconds": 207.38207828199666, "peak_cuda_mib": 2769.76025390625}
|
||||
{"step": 20, "epoch": 1, "loss": 0.10540554700855864, "seconds": 216.67654244400183, "peak_cuda_mib": 2769.76025390625}
|
||||
{"step": 21, "epoch": 1, "loss": 0.07820781783448183, "seconds": 226.29638538799918, "peak_cuda_mib": 2769.76025390625}
|
||||
{"step": 22, "epoch": 1, "loss": 0.07291922686999897, "seconds": 236.41520827299973, "peak_cuda_mib": 2769.76025390625}
|
||||
{"step": 23, "epoch": 1, "loss": 0.041554953380909865, "seconds": 246.51283356299973, "peak_cuda_mib": 2774.267578125}
|
||||
{"step": 24, "epoch": 1, "loss": 0.13455546938348562, "seconds": 255.15657514699706, "peak_cuda_mib": 2774.267578125}
|
||||
{"step": 25, "epoch": 1, "loss": 0.15407039164711023, "seconds": 263.5607540459969, "peak_cuda_mib": 2774.267578125, "validation_loss": 0.10121945928103042}
|
||||
{"step": 26, "epoch": 1, "loss": 0.1270278711890569, "seconds": 307.0034212649989, "peak_cuda_mib": 2774.267578125}
|
||||
{"step": 27, "epoch": 1, "loss": 0.12811397132099955, "seconds": 315.4646419819983, "peak_cuda_mib": 2774.267578125}
|
||||
{"step": 28, "epoch": 1, "loss": 0.14606271476804977, "seconds": 324.82921723600157, "peak_cuda_mib": 2774.267578125}
|
||||
{"step": 29, "epoch": 1, "loss": 0.049486552950838814, "seconds": 334.5532957350006, "peak_cuda_mib": 2774.267578125}
|
||||
{"step": 30, "epoch": 1, "loss": 0.11761647309322143, "seconds": 343.71294923200185, "peak_cuda_mib": 2774.267578125}
|
||||
{"step": 31, "epoch": 1, "loss": 0.08498146758574876, "seconds": 353.3541017559983, "peak_cuda_mib": 2774.267578125}
|
||||
{"step": 32, "epoch": 1, "loss": 0.1547634250491683, "seconds": 362.28539484400244, "peak_cuda_mib": 2774.267578125}
|
||||
{"step": 33, "epoch": 1, "loss": 0.1883373069576919, "seconds": 370.4769370099966, "peak_cuda_mib": 2774.267578125}
|
||||
{"step": 34, "epoch": 1, "loss": 0.10124442608503159, "seconds": 378.4543345689963, "peak_cuda_mib": 2774.267578125}
|
||||
{"step": 35, "epoch": 1, "loss": 0.0392077630640415, "seconds": 388.32370898299996, "peak_cuda_mib": 2774.267578125}
|
||||
{"step": 36, "epoch": 1, "loss": 0.10133314585254993, "seconds": 397.60372727399954, "peak_cuda_mib": 2774.267578125}
|
||||
{"step": 37, "epoch": 1, "loss": 0.031763357983436435, "seconds": 407.4506751700028, "peak_cuda_mib": 2774.267578125}
|
||||
{"step": 38, "epoch": 1, "loss": 0.12160474719712511, "seconds": 416.6591584170019, "peak_cuda_mib": 2774.267578125}
|
||||
{"step": 39, "epoch": 1, "loss": 0.08988130552461371, "seconds": 426.192453691001, "peak_cuda_mib": 2774.267578125}
|
||||
{"step": 40, "epoch": 1, "loss": 0.1025229657680029, "seconds": 435.6206649129963, "peak_cuda_mib": 2774.267578125}
|
||||
{"step": 41, "epoch": 1, "loss": 0.08306156097751227, "seconds": 445.8831847859983, "peak_cuda_mib": 2774.267578125}
|
||||
{"step": 42, "epoch": 1, "loss": 0.11939723963223514, "seconds": 454.4059897210027, "peak_cuda_mib": 2774.267578125}
|
||||
{"step": 43, "epoch": 1, "loss": 0.032355041617847746, "seconds": 463.75776703299925, "peak_cuda_mib": 2774.267578125}
|
||||
{"step": 44, "epoch": 1, "loss": 0.09126606645077118, "seconds": 474.65721592000045, "peak_cuda_mib": 2774.267578125}
|
||||
{"step": 45, "epoch": 1, "loss": 0.10815986017041723, "seconds": 483.9289711780002, "peak_cuda_mib": 2774.267578125}
|
||||
{"step": 46, "epoch": 1, "loss": 0.08192804168356815, "seconds": 492.7357436050006, "peak_cuda_mib": 2774.267578125}
|
||||
{"step": 47, "epoch": 1, "loss": 0.08134599040567991, "seconds": 502.8585111259963, "peak_cuda_mib": 2774.267578125}
|
||||
{"step": 48, "epoch": 1, "loss": 0.09908375598752173, "seconds": 512.6539711579971, "peak_cuda_mib": 2774.267578125}
|
||||
{"step": 49, "epoch": 1, "loss": 0.16285007508849958, "seconds": 520.8945014909987, "peak_cuda_mib": 2774.267578125}
|
||||
{"step": 50, "epoch": 1, "loss": 0.11961629847064614, "seconds": 529.2749978689972, "peak_cuda_mib": 2774.267578125, "validation_loss": 0.09914516313941792}
|
||||
{"step": 51, "epoch": 1, "loss": 0.06188828940685198, "seconds": 573.0266617969974, "peak_cuda_mib": 2774.267578125}
|
||||
{"step": 52, "epoch": 1, "loss": 0.07660351475169591, "seconds": 582.6575327070022, "peak_cuda_mib": 2774.267578125}
|
||||
{"step": 53, "epoch": 1, "loss": 0.1429038778951508, "seconds": 591.7470457200034, "peak_cuda_mib": 2774.267578125}
|
||||
{"step": 54, "epoch": 1, "loss": 0.0876359209905786, "seconds": 601.0829639980002, "peak_cuda_mib": 2774.267578125}
|
||||
{"step": 55, "epoch": 1, "loss": 0.0774427152100543, "seconds": 610.3920935049973, "peak_cuda_mib": 2774.267578125}
|
||||
{"step": 56, "epoch": 1, "loss": 0.06275974823074648, "seconds": 620.2322062320018, "peak_cuda_mib": 2774.267578125}
|
||||
{"step": 57, "epoch": 1, "loss": 0.059790863164380426, "seconds": 629.0907443179967, "peak_cuda_mib": 2774.267578125}
|
||||
{"step": 58, "epoch": 1, "loss": 0.04323218821082264, "seconds": 639.2130806019995, "peak_cuda_mib": 2774.267578125}
|
||||
{"step": 59, "epoch": 1, "loss": 0.03359975780767854, "seconds": 648.5642361790015, "peak_cuda_mib": 2774.267578125}
|
||||
{"step": 60, "epoch": 1, "loss": 0.09401738969609141, "seconds": 657.085457344001, "peak_cuda_mib": 2774.267578125}
|
||||
{"step": 61, "epoch": 1, "loss": 0.10420653857727302, "seconds": 666.4490791639982, "peak_cuda_mib": 2774.267578125}
|
||||
{"step": 62, "epoch": 1, "loss": 0.06494034207935329, "seconds": 674.8515353969997, "peak_cuda_mib": 2774.267578125}
|
||||
{"step": 63, "epoch": 1, "loss": 0.08578400554506516, "seconds": 684.3544385269997, "peak_cuda_mib": 2779.103515625}
|
||||
{"step": 64, "epoch": 1, "loss": 0.08956868508812477, "seconds": 692.9328568659985, "peak_cuda_mib": 2779.103515625}
|
||||
{"step": 65, "epoch": 1, "loss": 0.07740566666325321, "seconds": 702.3445224300012, "peak_cuda_mib": 2779.103515625}
|
||||
{"step": 66, "epoch": 1, "loss": 0.09748204581319442, "seconds": 712.9150303740025, "peak_cuda_mib": 2779.103515625}
|
||||
{"step": 67, "epoch": 1, "loss": 0.09408253024230362, "seconds": 722.3477353779963, "peak_cuda_mib": 2779.103515625}
|
||||
{"step": 68, "epoch": 1, "loss": 0.1544838160116342, "seconds": 732.0609969950019, "peak_cuda_mib": 2779.103515625}
|
||||
{"step": 69, "epoch": 1, "loss": 0.06002850037475582, "seconds": 741.9220733979964, "peak_cuda_mib": 2779.103515625}
|
||||
{"step": 70, "epoch": 1, "loss": 0.04894617456011474, "seconds": 752.2522782489978, "peak_cuda_mib": 2779.103515625}
|
||||
{"step": 71, "epoch": 1, "loss": 0.0644364555409993, "seconds": 762.4026968949984, "peak_cuda_mib": 2779.103515625}
|
||||
{"step": 72, "epoch": 1, "loss": 0.11124623291289026, "seconds": 772.407010349998, "peak_cuda_mib": 2779.103515625}
|
||||
{"step": 73, "epoch": 1, "loss": 0.09027093981603684, "seconds": 783.1766003379962, "peak_cuda_mib": 2779.103515625}
|
||||
{"step": 74, "epoch": 1, "loss": 0.12179316184483469, "seconds": 792.189860874998, "peak_cuda_mib": 2779.103515625}
|
||||
{"step": 75, "epoch": 1, "loss": 0.09010269505233737, "seconds": 800.8463720610016, "peak_cuda_mib": 2779.103515625, "validation_loss": 0.08720315554105931}
|
||||
{"step": 76, "epoch": 1, "loss": 0.0740406719269231, "seconds": 845.4801855810001, "peak_cuda_mib": 2779.103515625}
|
||||
{"step": 77, "epoch": 1, "loss": 0.08443864959917846, "seconds": 854.7814787059979, "peak_cuda_mib": 2779.103515625}
|
||||
{"step": 78, "epoch": 1, "loss": 0.09326180202879186, "seconds": 864.9992112320033, "peak_cuda_mib": 2779.103515625}
|
||||
{"step": 79, "epoch": 1, "loss": 0.10843068553731428, "seconds": 877.1159189600003, "peak_cuda_mib": 2779.103515625}
|
||||
{"step": 80, "epoch": 1, "loss": 0.09500505725281982, "seconds": 886.8208766730022, "peak_cuda_mib": 2779.103515625}
|
||||
{"step": 81, "epoch": 1, "loss": 0.0911472177831456, "seconds": 896.5897035789967, "peak_cuda_mib": 2779.103515625}
|
||||
{"step": 82, "epoch": 1, "loss": 0.0748484552877926, "seconds": 905.7747493990028, "peak_cuda_mib": 2779.103515625}
|
||||
{"step": 83, "epoch": 1, "loss": 0.11396347360459913, "seconds": 915.2009467850003, "peak_cuda_mib": 2779.103515625}
|
||||
{"step": 84, "epoch": 1, "loss": 0.09875626980829111, "seconds": 924.3261770449972, "peak_cuda_mib": 2779.103515625}
|
||||
{"step": 85, "epoch": 1, "loss": 0.21815292999963276, "seconds": 933.7138454660017, "peak_cuda_mib": 2779.103515625}
|
||||
{"step": 86, "epoch": 1, "loss": 0.09571092217993282, "seconds": 943.6202975409979, "peak_cuda_mib": 2779.103515625}
|
||||
{"step": 87, "epoch": 1, "loss": 0.10416132392401778, "seconds": 952.4447351619965, "peak_cuda_mib": 2779.103515625}
|
||||
{"step": 88, "epoch": 1, "loss": 0.12816633265038035, "seconds": 961.4652255920009, "peak_cuda_mib": 2779.103515625}
|
||||
{"step": 89, "epoch": 1, "loss": 0.10384324787355581, "seconds": 971.2473525519963, "peak_cuda_mib": 2779.103515625}
|
||||
{"step": 90, "epoch": 1, "loss": 0.082879335013331, "seconds": 980.9991367609982, "peak_cuda_mib": 2779.103515625}
|
||||
{"step": 91, "epoch": 1, "loss": 0.054579366992584255, "seconds": 990.7050228049993, "peak_cuda_mib": 2779.103515625}
|
||||
{"step": 92, "epoch": 1, "loss": 0.10663496636061609, "seconds": 1000.0718990530004, "peak_cuda_mib": 2779.103515625}
|
||||
{"step": 93, "epoch": 1, "loss": 0.07423936713166768, "seconds": 1009.1038821210022, "peak_cuda_mib": 2779.103515625}
|
||||
{"step": 94, "epoch": 1, "loss": 0.15296151553502568, "seconds": 1018.9726898230001, "peak_cuda_mib": 2779.103515625}
|
||||
{"step": 95, "epoch": 1, "loss": 0.07911808850803936, "seconds": 1028.9711100429995, "peak_cuda_mib": 2779.103515625}
|
||||
{"step": 96, "epoch": 1, "loss": 0.23531612257283996, "seconds": 1038.7752565050032, "peak_cuda_mib": 2779.103515625}
|
||||
{"step": 97, "epoch": 1, "loss": 0.06809451430262925, "seconds": 1048.7730714369973, "peak_cuda_mib": 2779.103515625}
|
||||
{"step": 98, "epoch": 1, "loss": 0.032844387896147964, "seconds": 1061.6314474480023, "peak_cuda_mib": 2779.103515625}
|
||||
{"step": 99, "epoch": 1, "loss": 0.11899201917503888, "seconds": 1078.9495686980008, "peak_cuda_mib": 2779.103515625}
|
||||
{"step": 100, "epoch": 1, "loss": 0.12370738689787686, "seconds": 1094.3222948280018, "peak_cuda_mib": 2779.103515625, "validation_loss": 0.08245812071940097}
|
||||
{"step": 101, "epoch": 1, "loss": 0.13659339305013418, "seconds": 1144.2924097029972, "peak_cuda_mib": 2779.103515625}
|
||||
{"step": 102, "epoch": 1, "loss": 0.061691983726632316, "seconds": 1153.670793641002, "peak_cuda_mib": 2779.103515625}
|
||||
{"step": 103, "epoch": 1, "loss": 0.13800899268244393, "seconds": 1162.7734558849988, "peak_cuda_mib": 2779.103515625}
|
||||
{"step": 104, "epoch": 1, "loss": 0.02434602929133689, "seconds": 1165.4224593530016, "peak_cuda_mib": 2779.103515625, "validation_loss": 0.08445217720992122}
|
||||
@@ -0,0 +1,25 @@
|
||||
{
|
||||
"kind": "executed_oracle_demonstrations",
|
||||
"seed": 42,
|
||||
"limitations": "Three-file synthetic repositories; query-word searches and oracle-chosen target ranges. Teaches protocol, not realistic repository planning.",
|
||||
"excluded_repositories_by_name": [
|
||||
"click",
|
||||
"flask",
|
||||
"requests"
|
||||
],
|
||||
"splits": {
|
||||
"train": {
|
||||
"trajectories": 256,
|
||||
"action_examples": 826,
|
||||
"source_sha256": "3eeed3185b74f934462aaccbd3c567f8db20754a1bf6f5f3686779f8b39f4338",
|
||||
"output_sha256": "af0abde43f8bbebfb384885b021d316b26354f8d4d4d6a535528d4c91708b756"
|
||||
},
|
||||
"validation": {
|
||||
"trajectories": 32,
|
||||
"action_examples": 104,
|
||||
"source_sha256": "e1b934c33f12322e4a56d0d17a987966c40961b035e15939fd716b76d4e017a2",
|
||||
"output_sha256": "57c33bcef55832c4e86e4cbc37de31aeb94d8f118b5389dbcb73fd09cd7fd264"
|
||||
}
|
||||
},
|
||||
"repository_disjoint": true
|
||||
}
|
||||
@@ -0,0 +1,92 @@
|
||||
{
|
||||
"base_id": "openbmb/MiniCPM5-1B",
|
||||
"base_revision": "87179e5c1f455ef22e6223592d2d61351b525bfc",
|
||||
"settings": {
|
||||
"data": "data/live-policy-v4",
|
||||
"output": "runs/minicpm5-policy-v1",
|
||||
"epochs": 1,
|
||||
"max_length": 2048,
|
||||
"learning_rate": 0.0001,
|
||||
"max_steps": 0
|
||||
},
|
||||
"examples": {
|
||||
"train": 826,
|
||||
"validation": 104
|
||||
},
|
||||
"overlength_dropped": {
|
||||
"train": 0,
|
||||
"validation": 0
|
||||
},
|
||||
"data_sha256": {
|
||||
"train": "af0abde43f8bbebfb384885b021d316b26354f8d4d4d6a535528d4c91708b756",
|
||||
"validation": "57c33bcef55832c4e86e4cbc37de31aeb94d8f118b5389dbcb73fd09cd7fd264"
|
||||
},
|
||||
"seed": 42,
|
||||
"batch_size": 1,
|
||||
"gradient_accumulation": 8,
|
||||
"lora_rank": 16,
|
||||
"lora_alpha": 32,
|
||||
"quantization": "nf4_double_quant",
|
||||
"loss": "assistant XML action tokens and end-of-turn token only",
|
||||
"data_manifest": {
|
||||
"kind": "executed_oracle_demonstrations",
|
||||
"seed": 42,
|
||||
"limitations": "Three-file synthetic repositories; query-word searches and oracle-chosen target ranges. Teaches protocol, not realistic repository planning.",
|
||||
"excluded_repositories_by_name": [
|
||||
"click",
|
||||
"flask",
|
||||
"requests"
|
||||
],
|
||||
"splits": {
|
||||
"train": {
|
||||
"trajectories": 256,
|
||||
"action_examples": 826,
|
||||
"source_sha256": "3eeed3185b74f934462aaccbd3c567f8db20754a1bf6f5f3686779f8b39f4338",
|
||||
"output_sha256": "af0abde43f8bbebfb384885b021d316b26354f8d4d4d6a535528d4c91708b756"
|
||||
},
|
||||
"validation": {
|
||||
"trajectories": 32,
|
||||
"action_examples": 104,
|
||||
"source_sha256": "e1b934c33f12322e4a56d0d17a987966c40961b035e15939fd716b76d4e017a2",
|
||||
"output_sha256": "57c33bcef55832c4e86e4cbc37de31aeb94d8f118b5389dbcb73fd09cd7fd264"
|
||||
}
|
||||
},
|
||||
"repository_disjoint": true
|
||||
},
|
||||
"packages": {
|
||||
"torch": "2.7.1",
|
||||
"transformers": "4.57.6",
|
||||
"peft": "0.17.1",
|
||||
"accelerate": "1.10.1",
|
||||
"bitsandbytes": "0.47.0"
|
||||
},
|
||||
"gpu": "NVIDIA GeForce RTX 3050 Laptop GPU",
|
||||
"source_sha256": {
|
||||
"__init__.py": "dcd2b573883b8068e806e3052adf9c03728f1cf621d35ff013179de76f76c702",
|
||||
"text.py": "8db7b80ee446480175f7691872303c8b72fd014604ad4fb575b359cee71f1938",
|
||||
"io.py": "b29c21db767ea9773281b17fe0754375153d18fb64f314907528f00d584a047f",
|
||||
"data.py": "20a891eff11d555d2301b62f4953146603efc87b03da4802f136f694ebdba78d",
|
||||
"encoder.py": "cb5671c08f28a70ffd0e2a7033528efc3581896a27242d55eb250ce922ab038f",
|
||||
"metrics.py": "cb93d30878c0c1006f06d1d0c4cbb188f90c6f4fc65f69f9a3b11d847f28ad27",
|
||||
"train.py": "cf20f6189967cfc88f3849bc3a225f2680c42026d53f204d9d5afc05fe88a2d6",
|
||||
"lexical.py": "6c47c834c6bbf7b317e2514057eee7444720581f5ec452ecb01ac666397b86b3",
|
||||
"symbols.py": "a377d6aa2c6b479c2ee115dd8e4859f0c06d188ae97b86fa1c7133885ee32d86",
|
||||
"index.py": "e8b0a666b41fed48d98616d5dcdd104e59c4308ac990281b463e5ee68f97877f",
|
||||
"evaluate.py": "29c942b9cbfff21ba8514511e27752527f0ec5e2cad5fbf57ec3b220fc7acd0b",
|
||||
"__main__.py": "ab30d9b696e41ece67094e19c87b1a31fc0ebce8d3afa79dc8eaf561f8fb4895",
|
||||
"download.py": "24f9802fb7cc82b81f6b1cb93e9244352d7b86feeb0a54ec055fb3e8d0a6a4d7",
|
||||
"cli.py": "d081962fc459808c65bd4c1d389e12ad9a9466df0a1374c8095a8bd5ccbb369e",
|
||||
"scout.py": "c3150baae51f37646305fee1d2c61e7b37f2ec9c4a0ba7f92ab4d7d6b1710161",
|
||||
"server.py": "483fe8051012249e38a6836cbe217d8c581d033e4d19026a29a7bd5c83878190",
|
||||
"live_tools.py": "e10fb6aaeaa6d7c4a5a35c0c6a210be4a4c283b18d215767d6bee0bac707d14c",
|
||||
"local_policy.py": "b0711b46a6399dfbeeecdcc045fb7392030e187d7e21c6e4fa9b51e6fc8056f4",
|
||||
"agent.py": "47357a900e9696b7ec5268eb031067eb0069faaeef92fd3159bd822d6bc67cdb",
|
||||
"live_server.py": "ccfb3c63115fe991c2f5f22c05f794d89c90d783b52f8ddbb668a8541e3af6fe",
|
||||
"native_protocol.py": "3a4d16c1d2cd90cb7d81e635f40d721270b0d541f454557a6d14e80b6b1073a7",
|
||||
"eval_live.py": "74946cf6ca0585810960f2a745261dd74e141d1342d7c33bc8db4c6ab34e2161",
|
||||
"prepare_live.py": "0675173c5e9c116c995c271dc06de914388f19eb83779c07e3009000a810290b",
|
||||
"live_data.py": "fd0bd5b118b2437146e08ef921dc4aa8a1da24e69bcbdd3efc5ff6e9c4fe8a78",
|
||||
"transformers_policy.py": "e8e8a81b70eb37563973e640534b425a7ea9fc7d398a39dc8d8f71b9be3f1b89",
|
||||
"train_policy.py": "803f45cb78087d1899c59bc09b6589e2ac84753c8f4be9938d837bbb5db3def8"
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,9 @@
|
||||
{
|
||||
"optimizer_steps": 104,
|
||||
"trainable_parameters": 11206656,
|
||||
"training_seconds": 1205.9964395429997,
|
||||
"best_validation_loss": 0.08245812071940097,
|
||||
"peak_cuda_allocated_mib": 2779.103515625,
|
||||
"selected_adapter": "runs/minicpm5-policy-v1/best",
|
||||
"note": "Validation measures teacher-forced actions on synthetic repositories, not task success."
|
||||
}
|
||||
@@ -0,0 +1,105 @@
|
||||
{"step": 0, "validation_loss": 0.4850238428379481, "seconds": 39.45231440700445}
|
||||
{"step": 1, "epoch": 1, "loss": 0.45637257769703865, "seconds": 49.92080000200076, "peak_cuda_mib": 2649.833984375}
|
||||
{"step": 2, "epoch": 1, "loss": 0.482717489823699, "seconds": 60.2498103570033, "peak_cuda_mib": 2752.13232421875}
|
||||
{"step": 3, "epoch": 1, "loss": 0.284512591548264, "seconds": 71.56246854100027, "peak_cuda_mib": 2792.00732421875}
|
||||
{"step": 4, "epoch": 1, "loss": 0.3362895091995597, "seconds": 80.65821186600078, "peak_cuda_mib": 2792.00732421875}
|
||||
{"step": 5, "epoch": 1, "loss": 0.2684406703338027, "seconds": 90.77562139300426, "peak_cuda_mib": 2792.00732421875}
|
||||
{"step": 6, "epoch": 1, "loss": 0.2656557112932205, "seconds": 100.38559429100133, "peak_cuda_mib": 2792.00732421875}
|
||||
{"step": 7, "epoch": 1, "loss": 0.1854343507438898, "seconds": 111.97682616700331, "peak_cuda_mib": 2800.2802734375}
|
||||
{"step": 8, "epoch": 1, "loss": 0.29348266031593084, "seconds": 120.8457516500057, "peak_cuda_mib": 2800.2802734375}
|
||||
{"step": 9, "epoch": 1, "loss": 0.15682652487885207, "seconds": 130.07469133500126, "peak_cuda_mib": 2800.2802734375}
|
||||
{"step": 10, "epoch": 1, "loss": 0.1607358451001346, "seconds": 141.46862508300546, "peak_cuda_mib": 2800.2802734375}
|
||||
{"step": 11, "epoch": 1, "loss": 0.17054182151332498, "seconds": 150.3780719800052, "peak_cuda_mib": 2800.2802734375}
|
||||
{"step": 12, "epoch": 1, "loss": 0.17258035857230425, "seconds": 160.67837398900156, "peak_cuda_mib": 2800.2802734375}
|
||||
{"step": 13, "epoch": 1, "loss": 0.2110563050955534, "seconds": 171.37938919899898, "peak_cuda_mib": 2808.56640625}
|
||||
{"step": 14, "epoch": 1, "loss": 0.22365222708322108, "seconds": 180.0792546360026, "peak_cuda_mib": 2808.56640625}
|
||||
{"step": 15, "epoch": 1, "loss": 0.18687963485717773, "seconds": 189.34065534600086, "peak_cuda_mib": 2808.56640625}
|
||||
{"step": 16, "epoch": 1, "loss": 0.1663389706518501, "seconds": 199.7823210600036, "peak_cuda_mib": 2808.56640625}
|
||||
{"step": 17, "epoch": 1, "loss": 0.2060576118528843, "seconds": 208.87418406000506, "peak_cuda_mib": 2808.56640625}
|
||||
{"step": 18, "epoch": 1, "loss": 0.14924282673746347, "seconds": 219.92342225700122, "peak_cuda_mib": 2813.3681640625}
|
||||
{"step": 19, "epoch": 1, "loss": 0.15162948216311634, "seconds": 230.1433086360048, "peak_cuda_mib": 2813.3681640625}
|
||||
{"step": 20, "epoch": 1, "loss": 0.17662409832701087, "seconds": 240.06778395600122, "peak_cuda_mib": 2813.3681640625}
|
||||
{"step": 21, "epoch": 1, "loss": 0.14288226701319218, "seconds": 249.6171102360022, "peak_cuda_mib": 2813.3681640625}
|
||||
{"step": 22, "epoch": 1, "loss": 0.15415711340028793, "seconds": 259.40911914499884, "peak_cuda_mib": 2813.3681640625}
|
||||
{"step": 23, "epoch": 1, "loss": 0.10877608216833323, "seconds": 270.7661820130015, "peak_cuda_mib": 2813.3681640625}
|
||||
{"step": 24, "epoch": 1, "loss": 0.07040677921031602, "seconds": 283.7168572760056, "peak_cuda_mib": 2813.3681640625}
|
||||
{"step": 25, "epoch": 1, "loss": 0.06180519983172417, "seconds": 293.7684336050006, "peak_cuda_mib": 2813.3681640625, "validation_loss": 0.12158928607599452}
|
||||
{"step": 26, "epoch": 1, "loss": 0.24939064076170325, "seconds": 340.9582841520023, "peak_cuda_mib": 2813.3681640625}
|
||||
{"step": 27, "epoch": 1, "loss": 0.18250937247648835, "seconds": 350.3378638530048, "peak_cuda_mib": 2813.3681640625}
|
||||
{"step": 28, "epoch": 1, "loss": 0.17073044972494245, "seconds": 360.79498653200426, "peak_cuda_mib": 2819.9130859375}
|
||||
{"step": 29, "epoch": 1, "loss": 0.23098348220810294, "seconds": 369.85720696900535, "peak_cuda_mib": 2819.9130859375}
|
||||
{"step": 30, "epoch": 1, "loss": 0.07457652446464635, "seconds": 381.4203124650012, "peak_cuda_mib": 2819.9130859375}
|
||||
{"step": 31, "epoch": 1, "loss": 0.1132696345448494, "seconds": 392.6046665680042, "peak_cuda_mib": 2819.9130859375}
|
||||
{"step": 32, "epoch": 1, "loss": 0.09791117676650174, "seconds": 402.79529738500423, "peak_cuda_mib": 2819.9130859375}
|
||||
{"step": 33, "epoch": 1, "loss": 0.12026501893706154, "seconds": 412.57553938800265, "peak_cuda_mib": 2819.9130859375}
|
||||
{"step": 34, "epoch": 1, "loss": 0.13367411214858294, "seconds": 421.6316583270018, "peak_cuda_mib": 2819.9130859375}
|
||||
{"step": 35, "epoch": 1, "loss": 0.08615786655718694, "seconds": 433.3629939020029, "peak_cuda_mib": 2819.9130859375}
|
||||
{"step": 36, "epoch": 1, "loss": 0.03197400968201691, "seconds": 445.2852147290032, "peak_cuda_mib": 2819.9130859375}
|
||||
{"step": 37, "epoch": 1, "loss": 0.058069856830115896, "seconds": 456.16411230000085, "peak_cuda_mib": 2819.9130859375}
|
||||
{"step": 38, "epoch": 1, "loss": 0.0692264427561895, "seconds": 468.8572670040012, "peak_cuda_mib": 2819.9130859375}
|
||||
{"step": 39, "epoch": 1, "loss": 0.06890509507502429, "seconds": 478.7925263360012, "peak_cuda_mib": 2819.9130859375}
|
||||
{"step": 40, "epoch": 1, "loss": 0.10617466503754258, "seconds": 487.7816622070022, "peak_cuda_mib": 2819.9130859375}
|
||||
{"step": 41, "epoch": 1, "loss": 0.13075172329263296, "seconds": 497.494467909004, "peak_cuda_mib": 2819.9130859375}
|
||||
{"step": 42, "epoch": 1, "loss": 0.09689546370645985, "seconds": 508.65737996200187, "peak_cuda_mib": 2819.9130859375}
|
||||
{"step": 43, "epoch": 1, "loss": 0.04528018506243825, "seconds": 519.0294696990022, "peak_cuda_mib": 2820.34326171875}
|
||||
{"step": 44, "epoch": 1, "loss": 0.12662469339556992, "seconds": 529.1242172120037, "peak_cuda_mib": 2820.34326171875}
|
||||
{"step": 45, "epoch": 1, "loss": 0.0609382136426575, "seconds": 539.6567460350052, "peak_cuda_mib": 2820.34326171875}
|
||||
{"step": 46, "epoch": 1, "loss": 0.06608481029979885, "seconds": 549.5729062760001, "peak_cuda_mib": 2820.34326171875}
|
||||
{"step": 47, "epoch": 1, "loss": 0.14198811282403767, "seconds": 560.0044448280023, "peak_cuda_mib": 2820.34326171875}
|
||||
{"step": 48, "epoch": 1, "loss": 0.028050338889443083, "seconds": 572.0866222850018, "peak_cuda_mib": 2820.34326171875}
|
||||
{"step": 49, "epoch": 1, "loss": 0.09169282211223617, "seconds": 581.8540288690056, "peak_cuda_mib": 2820.34326171875}
|
||||
{"step": 50, "epoch": 1, "loss": 0.12175777356605977, "seconds": 592.1475775549989, "peak_cuda_mib": 2820.34326171875, "validation_loss": 0.08274679601065425}
|
||||
{"step": 51, "epoch": 1, "loss": 0.03206814870645758, "seconds": 640.876971689002, "peak_cuda_mib": 2820.34326171875}
|
||||
{"step": 52, "epoch": 1, "loss": 0.1726230330823455, "seconds": 650.9458553860022, "peak_cuda_mib": 2820.34326171875}
|
||||
{"step": 53, "epoch": 1, "loss": 0.13705516411573626, "seconds": 661.8823420660046, "peak_cuda_mib": 2820.34326171875}
|
||||
{"step": 54, "epoch": 1, "loss": 0.1159228393516969, "seconds": 671.1677888330014, "peak_cuda_mib": 2820.34326171875}
|
||||
{"step": 55, "epoch": 1, "loss": 0.09259849619957095, "seconds": 684.3105562609999, "peak_cuda_mib": 2820.34326171875}
|
||||
{"step": 56, "epoch": 1, "loss": 0.10509921688208124, "seconds": 695.3493061879999, "peak_cuda_mib": 2820.34326171875}
|
||||
{"step": 57, "epoch": 1, "loss": 0.08911727752092702, "seconds": 705.8475802559988, "peak_cuda_mib": 2820.34326171875}
|
||||
{"step": 58, "epoch": 1, "loss": 0.04063783246965613, "seconds": 717.2269132970032, "peak_cuda_mib": 2820.34326171875}
|
||||
{"step": 59, "epoch": 1, "loss": 0.07510175523202633, "seconds": 728.1035653690051, "peak_cuda_mib": 2829.16748046875}
|
||||
{"step": 60, "epoch": 1, "loss": 0.06744589185109362, "seconds": 740.0061425810054, "peak_cuda_mib": 2829.16748046875}
|
||||
{"step": 61, "epoch": 1, "loss": 0.07332517191389343, "seconds": 749.9084139690021, "peak_cuda_mib": 2829.16748046875}
|
||||
{"step": 62, "epoch": 1, "loss": 0.013881501217838377, "seconds": 762.481225389005, "peak_cuda_mib": 2829.16748046875}
|
||||
{"step": 63, "epoch": 1, "loss": 0.06108983411104418, "seconds": 774.0962506729993, "peak_cuda_mib": 2829.16748046875}
|
||||
{"step": 64, "epoch": 1, "loss": 0.0714350834605284, "seconds": 783.2489020080029, "peak_cuda_mib": 2829.16748046875}
|
||||
{"step": 65, "epoch": 1, "loss": 0.039362836847431026, "seconds": 793.4984930100036, "peak_cuda_mib": 2829.16748046875}
|
||||
{"step": 66, "epoch": 1, "loss": 0.05296347808325663, "seconds": 805.674109986001, "peak_cuda_mib": 2829.16748046875}
|
||||
{"step": 67, "epoch": 1, "loss": 0.06943278170365375, "seconds": 815.6929812849994, "peak_cuda_mib": 2829.16748046875}
|
||||
{"step": 68, "epoch": 1, "loss": 0.14022759534418583, "seconds": 825.4839369030014, "peak_cuda_mib": 2829.16748046875}
|
||||
{"step": 69, "epoch": 1, "loss": 0.0425323024901445, "seconds": 834.8618998780003, "peak_cuda_mib": 2829.16748046875}
|
||||
{"step": 70, "epoch": 1, "loss": 0.053926261520246044, "seconds": 845.569591710002, "peak_cuda_mib": 2829.16748046875}
|
||||
{"step": 71, "epoch": 1, "loss": 0.08021661563543603, "seconds": 855.5936453030008, "peak_cuda_mib": 2829.16748046875}
|
||||
{"step": 72, "epoch": 1, "loss": 0.0861693425104022, "seconds": 866.3071101530004, "peak_cuda_mib": 2829.16748046875}
|
||||
{"step": 73, "epoch": 1, "loss": 0.11118061708111782, "seconds": 877.7371649519991, "peak_cuda_mib": 2829.16748046875}
|
||||
{"step": 74, "epoch": 1, "loss": 0.017422851771698333, "seconds": 887.0020884250043, "peak_cuda_mib": 2829.16748046875}
|
||||
{"step": 75, "epoch": 1, "loss": 0.13078330061398447, "seconds": 895.8012764890009, "peak_cuda_mib": 2829.16748046875, "validation_loss": 0.07012705635721571}
|
||||
{"step": 76, "epoch": 1, "loss": 0.08977905692881905, "seconds": 943.250254507002, "peak_cuda_mib": 2829.16748046875}
|
||||
{"step": 77, "epoch": 1, "loss": 0.07028561612241901, "seconds": 954.254906410999, "peak_cuda_mib": 2829.16748046875}
|
||||
{"step": 78, "epoch": 1, "loss": 0.06239214173183427, "seconds": 964.7154186960033, "peak_cuda_mib": 2829.16748046875}
|
||||
{"step": 79, "epoch": 1, "loss": 0.027898661130166147, "seconds": 973.9761887630011, "peak_cuda_mib": 2829.16748046875}
|
||||
{"step": 80, "epoch": 1, "loss": 0.07441900872800034, "seconds": 983.0644752580047, "peak_cuda_mib": 2829.16748046875}
|
||||
{"step": 81, "epoch": 1, "loss": 0.033230686120077735, "seconds": 993.7461950040015, "peak_cuda_mib": 2829.16748046875}
|
||||
{"step": 82, "epoch": 1, "loss": 0.013950775366538437, "seconds": 1004.6616329390017, "peak_cuda_mib": 2829.16748046875}
|
||||
{"step": 83, "epoch": 1, "loss": 0.10005052563064964, "seconds": 1014.7693986510058, "peak_cuda_mib": 2829.16748046875}
|
||||
{"step": 84, "epoch": 1, "loss": 0.027669295599480392, "seconds": 1025.4328220980024, "peak_cuda_mib": 2829.16748046875}
|
||||
{"step": 85, "epoch": 1, "loss": 0.024708810471565812, "seconds": 1035.536642813, "peak_cuda_mib": 2829.16748046875}
|
||||
{"step": 86, "epoch": 1, "loss": 0.046989022395791835, "seconds": 1046.250969351, "peak_cuda_mib": 2829.16748046875}
|
||||
{"step": 87, "epoch": 1, "loss": 0.0683449722964724, "seconds": 1055.5782757630004, "peak_cuda_mib": 2829.16748046875}
|
||||
{"step": 88, "epoch": 1, "loss": 0.02799615852200077, "seconds": 1065.3667068790019, "peak_cuda_mib": 2829.16748046875}
|
||||
{"step": 89, "epoch": 1, "loss": 0.04936668868685956, "seconds": 1075.226163195999, "peak_cuda_mib": 2829.16748046875}
|
||||
{"step": 90, "epoch": 1, "loss": 0.0372346756485058, "seconds": 1083.7401789130017, "peak_cuda_mib": 2829.16748046875}
|
||||
{"step": 91, "epoch": 1, "loss": 0.04032767558055639, "seconds": 1094.3077802430053, "peak_cuda_mib": 2842.462890625}
|
||||
{"step": 92, "epoch": 1, "loss": 0.04148191870081064, "seconds": 1104.0049289279996, "peak_cuda_mib": 2842.462890625}
|
||||
{"step": 93, "epoch": 1, "loss": 0.07540792894951664, "seconds": 1115.5138501300025, "peak_cuda_mib": 2842.462890625}
|
||||
{"step": 94, "epoch": 1, "loss": 0.05669689249316434, "seconds": 1127.4655595959994, "peak_cuda_mib": 2842.462890625}
|
||||
{"step": 95, "epoch": 1, "loss": 0.044251372943108436, "seconds": 1137.2640409170053, "peak_cuda_mib": 2842.462890625}
|
||||
{"step": 96, "epoch": 1, "loss": 0.055515681040560594, "seconds": 1146.6502688010005, "peak_cuda_mib": 2842.462890625}
|
||||
{"step": 97, "epoch": 1, "loss": 0.05971497527207248, "seconds": 1158.0348287950037, "peak_cuda_mib": 2842.462890625}
|
||||
{"step": 98, "epoch": 1, "loss": 0.12951946296379901, "seconds": 1168.0352086190032, "peak_cuda_mib": 2842.462890625}
|
||||
{"step": 99, "epoch": 1, "loss": 0.05692880804053857, "seconds": 1177.9049042090046, "peak_cuda_mib": 2842.462890625}
|
||||
{"step": 100, "epoch": 1, "loss": 0.028729630404995987, "seconds": 1187.3176044830034, "peak_cuda_mib": 2842.462890625, "validation_loss": 0.06807700138978572}
|
||||
{"step": 101, "epoch": 1, "loss": 0.02569994348596083, "seconds": 1236.317593741005, "peak_cuda_mib": 2842.462890625}
|
||||
{"step": 102, "epoch": 1, "loss": 0.06606083227597992, "seconds": 1245.7989182200035, "peak_cuda_mib": 2842.462890625}
|
||||
{"step": 103, "epoch": 1, "loss": 0.02389698278693686, "seconds": 1256.7493834760025, "peak_cuda_mib": 2842.462890625}
|
||||
{"step": 104, "epoch": 1, "loss": 0.12587661296129227, "seconds": 1259.172317223005, "peak_cuda_mib": 2842.462890625, "validation_loss": 0.06405698412138988}
|
||||
@@ -0,0 +1,102 @@
|
||||
{
|
||||
"base_id": "openbmb/MiniCPM5-1B",
|
||||
"base_revision": "87179e5c1f455ef22e6223592d2d61351b525bfc",
|
||||
"settings": {
|
||||
"data": "data/live-policy-windows-v1",
|
||||
"output": "runs/minicpm5-policy-v2",
|
||||
"epochs": 1,
|
||||
"max_length": 2560,
|
||||
"learning_rate": 0.0001,
|
||||
"max_steps": 0
|
||||
},
|
||||
"examples": {
|
||||
"train": 826,
|
||||
"validation": 104
|
||||
},
|
||||
"overlength_dropped": {
|
||||
"train": 0,
|
||||
"validation": 0
|
||||
},
|
||||
"data_sha256": {
|
||||
"train": "7c6315531aa37e5de59c6944a425cb9cb700474551a62a3fc51dd58134649b7b",
|
||||
"validation": "ac270e077c3e65f5590c18f27a3dec1fd80c48aac6067bc81b0e05f9b2cb0bc1"
|
||||
},
|
||||
"seed": 42,
|
||||
"batch_size": 1,
|
||||
"gradient_accumulation": 8,
|
||||
"lora_rank": 16,
|
||||
"lora_alpha": 32,
|
||||
"quantization": "nf4_double_quant",
|
||||
"loss": "assistant XML action tokens and end-of-turn token only",
|
||||
"data_manifest": {
|
||||
"kind": "executed_oracle_demonstrations",
|
||||
"seed": 42,
|
||||
"limitations": "Three-file synthetic repositories; query-word searches and oracle-chosen target files and final ranges. Teaches protocol, not realistic planning.",
|
||||
"recipe": "read-windows",
|
||||
"read_window": {
|
||||
"lines_before_match": 40,
|
||||
"lines_after_match": 60
|
||||
},
|
||||
"functions_per_file": 2,
|
||||
"target_position": "randomized before or after a distractor function",
|
||||
"excluded_repositories_by_name": [
|
||||
"click",
|
||||
"flask",
|
||||
"requests"
|
||||
],
|
||||
"splits": {
|
||||
"train": {
|
||||
"trajectories": 256,
|
||||
"action_examples": 826,
|
||||
"unobservable_candidates_skipped": 1,
|
||||
"source_sha256": "3eeed3185b74f934462aaccbd3c567f8db20754a1bf6f5f3686779f8b39f4338",
|
||||
"output_sha256": "7c6315531aa37e5de59c6944a425cb9cb700474551a62a3fc51dd58134649b7b"
|
||||
},
|
||||
"validation": {
|
||||
"trajectories": 32,
|
||||
"action_examples": 104,
|
||||
"unobservable_candidates_skipped": 0,
|
||||
"source_sha256": "e1b934c33f12322e4a56d0d17a987966c40961b035e15939fd716b76d4e017a2",
|
||||
"output_sha256": "ac270e077c3e65f5590c18f27a3dec1fd80c48aac6067bc81b0e05f9b2cb0bc1"
|
||||
}
|
||||
},
|
||||
"repository_disjoint": true
|
||||
},
|
||||
"packages": {
|
||||
"torch": "2.7.1",
|
||||
"transformers": "4.57.6",
|
||||
"peft": "0.17.1",
|
||||
"accelerate": "1.10.1",
|
||||
"bitsandbytes": "0.47.0"
|
||||
},
|
||||
"gpu": "NVIDIA GeForce RTX 3050 Laptop GPU",
|
||||
"source_sha256": {
|
||||
"__init__.py": "dcd2b573883b8068e806e3052adf9c03728f1cf621d35ff013179de76f76c702",
|
||||
"text.py": "8db7b80ee446480175f7691872303c8b72fd014604ad4fb575b359cee71f1938",
|
||||
"io.py": "b29c21db767ea9773281b17fe0754375153d18fb64f314907528f00d584a047f",
|
||||
"data.py": "20a891eff11d555d2301b62f4953146603efc87b03da4802f136f694ebdba78d",
|
||||
"encoder.py": "cb5671c08f28a70ffd0e2a7033528efc3581896a27242d55eb250ce922ab038f",
|
||||
"metrics.py": "cb93d30878c0c1006f06d1d0c4cbb188f90c6f4fc65f69f9a3b11d847f28ad27",
|
||||
"train.py": "cf20f6189967cfc88f3849bc3a225f2680c42026d53f204d9d5afc05fe88a2d6",
|
||||
"lexical.py": "6c47c834c6bbf7b317e2514057eee7444720581f5ec452ecb01ac666397b86b3",
|
||||
"symbols.py": "a377d6aa2c6b479c2ee115dd8e4859f0c06d188ae97b86fa1c7133885ee32d86",
|
||||
"index.py": "e8b0a666b41fed48d98616d5dcdd104e59c4308ac990281b463e5ee68f97877f",
|
||||
"evaluate.py": "29c942b9cbfff21ba8514511e27752527f0ec5e2cad5fbf57ec3b220fc7acd0b",
|
||||
"__main__.py": "ab30d9b696e41ece67094e19c87b1a31fc0ebce8d3afa79dc8eaf561f8fb4895",
|
||||
"download.py": "24f9802fb7cc82b81f6b1cb93e9244352d7b86feeb0a54ec055fb3e8d0a6a4d7",
|
||||
"cli.py": "d081962fc459808c65bd4c1d389e12ad9a9466df0a1374c8095a8bd5ccbb369e",
|
||||
"scout.py": "c3150baae51f37646305fee1d2c61e7b37f2ec9c4a0ba7f92ab4d7d6b1710161",
|
||||
"server.py": "483fe8051012249e38a6836cbe217d8c581d033e4d19026a29a7bd5c83878190",
|
||||
"live_tools.py": "e10fb6aaeaa6d7c4a5a35c0c6a210be4a4c283b18d215767d6bee0bac707d14c",
|
||||
"local_policy.py": "b0711b46a6399dfbeeecdcc045fb7392030e187d7e21c6e4fa9b51e6fc8056f4",
|
||||
"agent.py": "47357a900e9696b7ec5268eb031067eb0069faaeef92fd3159bd822d6bc67cdb",
|
||||
"live_server.py": "ccfb3c63115fe991c2f5f22c05f794d89c90d783b52f8ddbb668a8541e3af6fe",
|
||||
"native_protocol.py": "3a4d16c1d2cd90cb7d81e635f40d721270b0d541f454557a6d14e80b6b1073a7",
|
||||
"eval_live.py": "38250d45fd7e60c23c4c86f5819242736f29df8af384f92713128c84ffd734ff",
|
||||
"prepare_live.py": "0675173c5e9c116c995c271dc06de914388f19eb83779c07e3009000a810290b",
|
||||
"live_data.py": "4d064db6c4da6ef3b55c63e92c0da4ceb62d5cecd746458a90e0b7a9b4e86913",
|
||||
"transformers_policy.py": "e8e8a81b70eb37563973e640534b425a7ea9fc7d398a39dc8d8f71b9be3f1b89",
|
||||
"train_policy.py": "d712111e3b7272372e68986f62c464afcb66d0c4b13b2cdd632459980e2091a8",
|
||||
"keyword_baseline.py": "5fe18c76b4c6e085c358ab5c490d717ffdd82b917a2882cf826fa26c57f64816"
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,9 @@
|
||||
{
|
||||
"optimizer_steps": 104,
|
||||
"trainable_parameters": 11206656,
|
||||
"training_seconds": 1297.2961119380052,
|
||||
"best_validation_loss": 0.06405698412138988,
|
||||
"peak_cuda_allocated_mib": 2842.462890625,
|
||||
"selected_adapter": "runs/minicpm5-policy-v2/best",
|
||||
"note": "Validation measures teacher-forced actions on synthetic repositories, not task success."
|
||||
}
|
||||
@@ -0,0 +1,489 @@
|
||||
{
|
||||
"suite": {
|
||||
"name": "live-search-v1",
|
||||
"scope": "30 hand-authored English single-function localization tasks in three public Python repositories. Development benchmark, not a downstream coding-task or contamination-free evaluation.",
|
||||
"repositories": {
|
||||
"requests": {
|
||||
"commit": "b25c87d7cb8d6a18a37fa12442b5f883f9e41741",
|
||||
"url": "https://github.com/psf/requests.git"
|
||||
},
|
||||
"flask": {
|
||||
"commit": "2c1b30d0503cfb064f1cb252e6614a06915a362a",
|
||||
"url": "https://github.com/pallets/flask.git"
|
||||
},
|
||||
"click": {
|
||||
"commit": "fd183b2ced1cb5857784fe7fb22f4982f671f098",
|
||||
"url": "https://github.com/pallets/click.git"
|
||||
}
|
||||
},
|
||||
"tasks": [
|
||||
{
|
||||
"id": "requests-01",
|
||||
"repository": "requests",
|
||||
"query": "Find where an unsuccessful HTTP status becomes an exception containing the server reason and URL.",
|
||||
"targets": [
|
||||
{
|
||||
"path": "src/requests/models.py",
|
||||
"symbol": "Response.raise_for_status",
|
||||
"start_line": 1002,
|
||||
"end_line": 1026,
|
||||
"sha256": "32365d67893bb67c3ed67cf93ca4a18e63e6ab29342fa0dc8b09c59e06ff564e"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "requests-02",
|
||||
"repository": "requests",
|
||||
"query": "Locate the response iterator that preserves an incomplete trailing line across downloaded chunks.",
|
||||
"targets": [
|
||||
{
|
||||
"path": "src/requests/models.py",
|
||||
"symbol": "Response.iter_lines",
|
||||
"start_line": 867,
|
||||
"end_line": 888,
|
||||
"sha256": "32365d67893bb67c3ed67cf93ca4a18e63e6ab29342fa0dc8b09c59e06ff564e"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "requests-03",
|
||||
"repository": "requests",
|
||||
"query": "Where is it decided whether credentials may survive a redirect to another hostname, port, or protocol?",
|
||||
"targets": [
|
||||
{
|
||||
"path": "src/requests/sessions.py",
|
||||
"symbol": "SessionRedirectMixin.should_strip_auth",
|
||||
"start_line": 129,
|
||||
"end_line": 157,
|
||||
"sha256": "0a5d5da449ce7f0af3ccf6e4bbe7a67a935e37846dff4ff9f08cb6c7e2464e6f"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "requests-04",
|
||||
"repository": "requests",
|
||||
"query": "Find the logic that changes the HTTP verb when following 301, 302, or 303 redirects.",
|
||||
"targets": [
|
||||
{
|
||||
"path": "src/requests/sessions.py",
|
||||
"symbol": "SessionRedirectMixin.rebuild_method",
|
||||
"start_line": 337,
|
||||
"end_line": 353,
|
||||
"sha256": "0a5d5da449ce7f0af3ccf6e4bbe7a67a935e37846dff4ff9f08cb6c7e2464e6f"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "requests-05",
|
||||
"repository": "requests",
|
||||
"query": "Locate where environment proxy settings and certificate bundle variables are merged with session options.",
|
||||
"targets": [
|
||||
{
|
||||
"path": "src/requests/sessions.py",
|
||||
"symbol": "Session.merge_environment_settings",
|
||||
"start_line": 757,
|
||||
"end_line": 779,
|
||||
"sha256": "0a5d5da449ce7f0af3ccf6e4bbe7a67a935e37846dff4ff9f08cb6c7e2464e6f"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "requests-06",
|
||||
"repository": "requests",
|
||||
"query": "Find where a 401 digest challenge causes the original request to be copied and sent again with authentication.",
|
||||
"targets": [
|
||||
{
|
||||
"path": "src/requests/auth.py",
|
||||
"symbol": "HTTPDigestAuth.handle_401",
|
||||
"start_line": 250,
|
||||
"end_line": 283,
|
||||
"sha256": "905ef9b6a9cb72d67d31ffe19bd4d9223e1c4169cde6ec51cfca16b31e70991d"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "requests-07",
|
||||
"repository": "requests",
|
||||
"query": "Locate the helper that fills a cookie container from a mapping while optionally preserving existing names.",
|
||||
"targets": [
|
||||
{
|
||||
"path": "src/requests/cookies.py",
|
||||
"symbol": "cookiejar_from_dict",
|
||||
"start_line": 530,
|
||||
"end_line": 539,
|
||||
"sha256": "6cd8be8aa123e0d3d9d34fa86feac7bf392f39bccdde5129830de0ea9692dd7c"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "requests-08",
|
||||
"repository": "requests",
|
||||
"query": "Find where credentials are loaded from the users netrc file, including NETRC and home-directory lookup.",
|
||||
"targets": [
|
||||
{
|
||||
"path": "src/requests/utils.py",
|
||||
"symbol": "get_netrc_auth",
|
||||
"start_line": 210,
|
||||
"end_line": 248,
|
||||
"sha256": "5aa53ceab677c2f842fad42359c8ed1ff1c4299c1607789609957a496e4311d4"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "requests-09",
|
||||
"repository": "requests",
|
||||
"query": "Find the implementation that seeks a request body back to its saved position and fails for an unrewindable stream.",
|
||||
"targets": [
|
||||
{
|
||||
"path": "src/requests/utils.py",
|
||||
"symbol": "rewind_body",
|
||||
"start_line": 1075,
|
||||
"end_line": 1086,
|
||||
"sha256": "5aa53ceab677c2f842fad42359c8ed1ff1c4299c1607789609957a496e4311d4"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "requests-10",
|
||||
"repository": "requests",
|
||||
"query": "Where is a request URL validated, its international hostname encoded, and its query parameters appended?",
|
||||
"targets": [
|
||||
{
|
||||
"path": "src/requests/models.py",
|
||||
"symbol": "PreparedRequest.prepare_url",
|
||||
"start_line": 416,
|
||||
"end_line": 481,
|
||||
"sha256": "32365d67893bb67c3ed67cf93ca4a18e63e6ab29342fa0dc8b09c59e06ff564e"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "flask-01",
|
||||
"repository": "flask",
|
||||
"query": "Find the code that invokes the view function selected by the matched URL rule.",
|
||||
"targets": [
|
||||
{
|
||||
"path": "src/flask/app.py",
|
||||
"symbol": "Flask.dispatch_request",
|
||||
"start_line": 889,
|
||||
"end_line": 902,
|
||||
"sha256": "5c6aa0151b0b8018732280761d27eda0c4c83378630e21faadd57b73783720a7"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "flask-02",
|
||||
"repository": "flask",
|
||||
"query": "Where are view return values such as tuples, dictionaries and strings converted into a response object?",
|
||||
"targets": [
|
||||
{
|
||||
"path": "src/flask/app.py",
|
||||
"symbol": "Flask.make_response",
|
||||
"start_line": 1186,
|
||||
"end_line": 1269,
|
||||
"sha256": "5c6aa0151b0b8018732280761d27eda0c4c83378630e21faadd57b73783720a7"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "flask-03",
|
||||
"repository": "flask",
|
||||
"query": "Find where coroutine view functions are adapted for synchronous request handling.",
|
||||
"targets": [
|
||||
{
|
||||
"path": "src/flask/app.py",
|
||||
"symbol": "Flask.ensure_sync",
|
||||
"start_line": 975,
|
||||
"end_line": 978,
|
||||
"sha256": "5c6aa0151b0b8018732280761d27eda0c4c83378630e21faadd57b73783720a7"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "flask-04",
|
||||
"repository": "flask",
|
||||
"query": "Locate the processing that runs after-request callbacks and saves the session before returning the response.",
|
||||
"targets": [
|
||||
{
|
||||
"path": "src/flask/app.py",
|
||||
"symbol": "Flask.process_response",
|
||||
"start_line": 1311,
|
||||
"end_line": 1324,
|
||||
"sha256": "5c6aa0151b0b8018732280761d27eda0c4c83378630e21faadd57b73783720a7"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "flask-05",
|
||||
"repository": "flask",
|
||||
"query": "Find where before-request handlers can short-circuit normal request dispatch by returning a value.",
|
||||
"targets": [
|
||||
{
|
||||
"path": "src/flask/app.py",
|
||||
"symbol": "Flask.preprocess_request",
|
||||
"start_line": 1281,
|
||||
"end_line": 1296,
|
||||
"sha256": "5c6aa0151b0b8018732280761d27eda0c4c83378630e21faadd57b73783720a7"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "flask-06",
|
||||
"repository": "flask",
|
||||
"query": "Where is a streaming generator wrapped so the request context remains active while producing its items?",
|
||||
"targets": [
|
||||
{
|
||||
"path": "src/flask/helpers.py",
|
||||
"symbol": "stream_with_context",
|
||||
"start_line": 104,
|
||||
"end_line": 143,
|
||||
"sha256": "ac96607bbfc9dbcf09d5442fe7e90d7f8a046b0df7d8f3fc3535b44112016d71"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "flask-07",
|
||||
"repository": "flask",
|
||||
"query": "Find where a categorized one-time message is stored in the session and a notification signal is emitted.",
|
||||
"targets": [
|
||||
{
|
||||
"path": "src/flask/helpers.py",
|
||||
"symbol": "flash",
|
||||
"start_line": 340,
|
||||
"end_line": 349,
|
||||
"sha256": "ac96607bbfc9dbcf09d5442fe7e90d7f8a046b0df7d8f3fc3535b44112016d71"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "flask-08",
|
||||
"repository": "flask",
|
||||
"query": "Locate where stored one-time messages are popped from the session, cached for the request, and filtered by category.",
|
||||
"targets": [
|
||||
{
|
||||
"path": "src/flask/helpers.py",
|
||||
"symbol": "get_flashed_messages",
|
||||
"start_line": 383,
|
||||
"end_line": 391,
|
||||
"sha256": "ac96607bbfc9dbcf09d5442fe7e90d7f8a046b0df7d8f3fc3535b44112016d71"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "flask-09",
|
||||
"repository": "flask",
|
||||
"query": "Find where the command-line startup reads dotenv files without replacing environment variables already set.",
|
||||
"targets": [
|
||||
{
|
||||
"path": "src/flask/cli.py",
|
||||
"symbol": "load_dotenv",
|
||||
"start_line": 740,
|
||||
"end_line": 771,
|
||||
"sha256": "3df87bdbe07196fa07d101c20ce7351ef5c6ecaab95deb5fdf3ccdcf690d4879"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "flask-10",
|
||||
"repository": "flask",
|
||||
"query": "Locate where a signed session cookie is verified and an invalid signature produces an empty session.",
|
||||
"targets": [
|
||||
{
|
||||
"path": "src/flask/sessions.py",
|
||||
"symbol": "SecureCookieSessionInterface.open_session",
|
||||
"start_line": 338,
|
||||
"end_line": 349,
|
||||
"sha256": "76ebd81a608687f1f772032ecb6a1e4a3ac2b02bcbca517808128e5e9374426b"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "click-01",
|
||||
"repository": "click",
|
||||
"query": "Find where a supplied option value is normalized and matched against the allowed choices.",
|
||||
"targets": [
|
||||
{
|
||||
"path": "src/click/types.py",
|
||||
"symbol": "Choice.convert",
|
||||
"start_line": 344,
|
||||
"end_line": 358,
|
||||
"sha256": "2814d1c4ddbc711d5567999bf62257f7c3104b0fe9f521b3963a9f108f33e53c"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "click-02",
|
||||
"repository": "click",
|
||||
"query": "Locate where a filesystem argument is checked for existence, readability, writability, and allowed file or directory type.",
|
||||
"targets": [
|
||||
{
|
||||
"path": "src/click/types.py",
|
||||
"symbol": "Path.convert",
|
||||
"start_line": 930,
|
||||
"end_line": 995,
|
||||
"sha256": "2814d1c4ddbc711d5567999bf62257f7c3104b0fe9f521b3963a9f108f33e53c"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "click-03",
|
||||
"repository": "click",
|
||||
"query": "Find where a date argument is parsed by trying several accepted formats and reports a failure if none match.",
|
||||
"targets": [
|
||||
{
|
||||
"path": "src/click/types.py",
|
||||
"symbol": "DateTime.convert",
|
||||
"start_line": 448,
|
||||
"end_line": 466,
|
||||
"sha256": "2814d1c4ddbc711d5567999bf62257f7c3104b0fe9f521b3963a9f108f33e53c"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "click-04",
|
||||
"repository": "click",
|
||||
"query": "Where is an option value from an environment variable split and grouped for multiple arguments?",
|
||||
"targets": [
|
||||
{
|
||||
"path": "src/click/core.py",
|
||||
"symbol": "Option.value_from_envvar",
|
||||
"start_line": 2983,
|
||||
"end_line": 2996,
|
||||
"sha256": "814869352f5c14119d1175dd8ac1951be791bc67f8f514f215a831ba5ab0745c"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "click-05",
|
||||
"repository": "click",
|
||||
"query": "Locate where a command group resolves a subcommand name, retries normalized names, and rejects unknown commands.",
|
||||
"targets": [
|
||||
{
|
||||
"path": "src/click/core.py",
|
||||
"symbol": "Group.resolve_command",
|
||||
"start_line": 1867,
|
||||
"end_line": 1889,
|
||||
"sha256": "814869352f5c14119d1175dd8ac1951be791bc67f8f514f215a831ba5ab0745c"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "click-06",
|
||||
"repository": "click",
|
||||
"query": "Find the helper that enters a context manager and registers its cleanup with the command context.",
|
||||
"targets": [
|
||||
{
|
||||
"path": "src/click/core.py",
|
||||
"symbol": "Context.with_resource",
|
||||
"start_line": 598,
|
||||
"end_line": 598,
|
||||
"sha256": "814869352f5c14119d1175dd8ac1951be791bc67f8f514f215a831ba5ab0745c"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "click-07",
|
||||
"repository": "click",
|
||||
"query": "Find where each element of a tuple argument is converted using its corresponding parameter type.",
|
||||
"targets": [
|
||||
{
|
||||
"path": "src/click/types.py",
|
||||
"symbol": "Tuple.convert",
|
||||
"start_line": 1049,
|
||||
"end_line": 1065,
|
||||
"sha256": "2814d1c4ddbc711d5567999bf62257f7c3104b0fe9f521b3963a9f108f33e53c"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "click-08",
|
||||
"repository": "click",
|
||||
"query": "Locate the interactive yes-or-no question loop that retries invalid answers and can abort on rejection.",
|
||||
"targets": [
|
||||
{
|
||||
"path": "src/click/termui.py",
|
||||
"symbol": "confirm",
|
||||
"start_line": 223,
|
||||
"end_line": 252,
|
||||
"sha256": "bc062b282d9aedffcd7c42210138445587dafff89be5741b4d2086b499400510"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "click-09",
|
||||
"repository": "click",
|
||||
"query": "Find the testing helper that temporarily changes the current directory and removes its temporary directory on exit.",
|
||||
"targets": [
|
||||
{
|
||||
"path": "src/click/testing.py",
|
||||
"symbol": "CliRunner.isolated_filesystem",
|
||||
"start_line": 552,
|
||||
"end_line": 565,
|
||||
"sha256": "d9e2dd01a0890864e83f94f0f9737c263ac36f7c1b2487c047183777472b5b93"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "click-10",
|
||||
"repository": "click",
|
||||
"query": "Where does a lazily opened file acquire its actual stream and translate operating-system errors into a file error?",
|
||||
"targets": [
|
||||
{
|
||||
"path": "src/click/utils.py",
|
||||
"symbol": "LazyFile.open",
|
||||
"start_line": 156,
|
||||
"end_line": 167,
|
||||
"sha256": "6f5326faeb040c11ed13070f96d3c89f7cd22b89f0d8a4e9e460bbfe849394ba"
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
},
|
||||
"suite_sha256": "d06effdf41ee2e38bf8f44629949d9b4491067986fc5a73aef22cf80c5a5bf96",
|
||||
"model": {
|
||||
"model": "openbmb/MiniCPM5-1B",
|
||||
"base_id": "openbmb/MiniCPM5-1B",
|
||||
"base_revision": "87179e5c1f455ef22e6223592d2d61351b525bfc",
|
||||
"backend": "transformers",
|
||||
"quantization": "nf4",
|
||||
"adapter": null,
|
||||
"load_seconds": 5.293721280999307
|
||||
},
|
||||
"source_sha256": {
|
||||
"__init__.py": "dcd2b573883b8068e806e3052adf9c03728f1cf621d35ff013179de76f76c702",
|
||||
"text.py": "8db7b80ee446480175f7691872303c8b72fd014604ad4fb575b359cee71f1938",
|
||||
"io.py": "b29c21db767ea9773281b17fe0754375153d18fb64f314907528f00d584a047f",
|
||||
"data.py": "20a891eff11d555d2301b62f4953146603efc87b03da4802f136f694ebdba78d",
|
||||
"encoder.py": "cb5671c08f28a70ffd0e2a7033528efc3581896a27242d55eb250ce922ab038f",
|
||||
"metrics.py": "cb93d30878c0c1006f06d1d0c4cbb188f90c6f4fc65f69f9a3b11d847f28ad27",
|
||||
"train.py": "cf20f6189967cfc88f3849bc3a225f2680c42026d53f204d9d5afc05fe88a2d6",
|
||||
"lexical.py": "6c47c834c6bbf7b317e2514057eee7444720581f5ec452ecb01ac666397b86b3",
|
||||
"symbols.py": "a377d6aa2c6b479c2ee115dd8e4859f0c06d188ae97b86fa1c7133885ee32d86",
|
||||
"index.py": "e8b0a666b41fed48d98616d5dcdd104e59c4308ac990281b463e5ee68f97877f",
|
||||
"evaluate.py": "29c942b9cbfff21ba8514511e27752527f0ec5e2cad5fbf57ec3b220fc7acd0b",
|
||||
"__main__.py": "ab30d9b696e41ece67094e19c87b1a31fc0ebce8d3afa79dc8eaf561f8fb4895",
|
||||
"download.py": "24f9802fb7cc82b81f6b1cb93e9244352d7b86feeb0a54ec055fb3e8d0a6a4d7",
|
||||
"cli.py": "d081962fc459808c65bd4c1d389e12ad9a9466df0a1374c8095a8bd5ccbb369e",
|
||||
"scout.py": "c3150baae51f37646305fee1d2c61e7b37f2ec9c4a0ba7f92ab4d7d6b1710161",
|
||||
"server.py": "483fe8051012249e38a6836cbe217d8c581d033e4d19026a29a7bd5c83878190",
|
||||
"live_tools.py": "e10fb6aaeaa6d7c4a5a35c0c6a210be4a4c283b18d215767d6bee0bac707d14c",
|
||||
"local_policy.py": "6d2b9dee7cf2688e5ff41b487151704839cfe2689f3c266905d3448573d985ad",
|
||||
"agent.py": "726ba5b78dba4e2b47d4e088789f3f767c92871b9c5fe44a3e4634722a5e516c",
|
||||
"live_server.py": "102fd679ac876d3ad63d65db5813eea4c2a7dfc4a6124f4c169a1327b3b7804b",
|
||||
"native_protocol.py": "3a4d16c1d2cd90cb7d81e635f40d721270b0d541f454557a6d14e80b6b1073a7",
|
||||
"eval_live.py": "74946cf6ca0585810960f2a745261dd74e141d1342d7c33bc8db4c6ab34e2161",
|
||||
"prepare_live.py": "072e53690f3113b260b57d39db2a863414995b895380e7149c621e9063f16f07",
|
||||
"live_data.py": "97b458cb24e2d782c715a849898e973ea3fe5cd4fb99e6ed9ea238a7244a0401",
|
||||
"transformers_policy.py": "e8e8a81b70eb37563973e640534b425a7ea9fc7d398a39dc8d8f71b9be3f1b89",
|
||||
"train_policy.py": "3087bbfbc995088ed2db8684cc8e3a721bc3c523895feb8fe917f0ac44f6ccea"
|
||||
},
|
||||
"max_rounds": 6,
|
||||
"max_chars": 6000,
|
||||
"timeout_seconds": 90,
|
||||
"context": 8192,
|
||||
"max_generation_tokens": 512,
|
||||
"tokenizer": "pinned_hf",
|
||||
"temperature": 0,
|
||||
"seed": 42,
|
||||
"scope": "Single-function localization; no large-model baseline; public repositories may have appeared in the base model's pretraining."
|
||||
}
|
||||
@@ -0,0 +1,88 @@
|
||||
{
|
||||
"tasks": 30,
|
||||
"target_hit_rate": 0,
|
||||
"file_hit_rate": 0.13333333333333333,
|
||||
"macro_line_precision": 0.0,
|
||||
"macro_line_recall": 0.0,
|
||||
"macro_line_f1": 0.0,
|
||||
"latency_median_seconds": 16.385701761500968,
|
||||
"latency_p95_seconds": 54.43225493735008,
|
||||
"statuses": {
|
||||
"budget_exhausted": 17,
|
||||
"completed": 13
|
||||
},
|
||||
"total_tool_errors": 16,
|
||||
"total_invalid_actions": 53,
|
||||
"mean_rounds": 5.233333333333333,
|
||||
"mean_tool_calls": 4.033333333333333,
|
||||
"total_input_tokens": 361812,
|
||||
"total_output_tokens": 7292,
|
||||
"mean_returned_chars": 224.76666666666668,
|
||||
"per_repository": {
|
||||
"requests": {
|
||||
"tasks": 10,
|
||||
"target_hit_rate": 0,
|
||||
"file_hit_rate": 0.2,
|
||||
"macro_line_precision": 0.0,
|
||||
"macro_line_recall": 0.0,
|
||||
"macro_line_f1": 0.0,
|
||||
"latency_median_seconds": 17.27006782099852,
|
||||
"latency_p95_seconds": 21.234492207000223,
|
||||
"statuses": {
|
||||
"budget_exhausted": 6,
|
||||
"completed": 4
|
||||
},
|
||||
"total_tool_errors": 10,
|
||||
"total_invalid_actions": 12,
|
||||
"mean_rounds": 5.6,
|
||||
"mean_tool_calls": 5,
|
||||
"total_input_tokens": 149348,
|
||||
"total_output_tokens": 1865,
|
||||
"mean_returned_chars": 259.1
|
||||
},
|
||||
"flask": {
|
||||
"tasks": 10,
|
||||
"target_hit_rate": 0,
|
||||
"file_hit_rate": 0.1,
|
||||
"macro_line_precision": 0.0,
|
||||
"macro_line_recall": 0.0,
|
||||
"macro_line_f1": 0.0,
|
||||
"latency_median_seconds": 14.9863461620007,
|
||||
"latency_p95_seconds": 57.56604749939506,
|
||||
"statuses": {
|
||||
"completed": 4,
|
||||
"budget_exhausted": 6
|
||||
},
|
||||
"total_tool_errors": 3,
|
||||
"total_invalid_actions": 29,
|
||||
"mean_rounds": 5,
|
||||
"mean_tool_calls": 2.7,
|
||||
"total_input_tokens": 107622,
|
||||
"total_output_tokens": 2444,
|
||||
"mean_returned_chars": 348.9
|
||||
},
|
||||
"click": {
|
||||
"tasks": 10,
|
||||
"target_hit_rate": 0,
|
||||
"file_hit_rate": 0.1,
|
||||
"macro_line_precision": 0.0,
|
||||
"macro_line_recall": 0.0,
|
||||
"macro_line_f1": 0.0,
|
||||
"latency_median_seconds": 16.385701761500968,
|
||||
"latency_p95_seconds": 54.43225493735008,
|
||||
"statuses": {
|
||||
"budget_exhausted": 5,
|
||||
"completed": 5
|
||||
},
|
||||
"total_tool_errors": 3,
|
||||
"total_invalid_actions": 12,
|
||||
"mean_rounds": 5.1,
|
||||
"mean_tool_calls": 4.4,
|
||||
"total_input_tokens": 104842,
|
||||
"total_output_tokens": 2983,
|
||||
"mean_returned_chars": 66.3
|
||||
}
|
||||
},
|
||||
"sampled_peak_gpu_memory_mib": 2198,
|
||||
"gpu_sampling_interval_seconds": 1
|
||||
}
|
||||
@@ -0,0 +1,796 @@
|
||||
[
|
||||
{
|
||||
"id": "requests-01",
|
||||
"repository": "requests",
|
||||
"scores": {
|
||||
"file_hit": false,
|
||||
"target_hit": false,
|
||||
"line_precision": 0.0,
|
||||
"line_recall": 0.0,
|
||||
"line_f1": 0.0,
|
||||
"returned_lines": 0,
|
||||
"overlap_lines": 0
|
||||
},
|
||||
"status": "budget_exhausted",
|
||||
"elapsed_seconds": 19.568732997999177,
|
||||
"rounds": 6,
|
||||
"tool_calls": 5,
|
||||
"tool_errors": 1,
|
||||
"invalid_actions": 2,
|
||||
"input_tokens": 19017,
|
||||
"output_tokens": 207,
|
||||
"returned_chars": 0,
|
||||
"references": []
|
||||
},
|
||||
{
|
||||
"id": "requests-02",
|
||||
"repository": "requests",
|
||||
"scores": {
|
||||
"file_hit": false,
|
||||
"target_hit": false,
|
||||
"line_precision": 0.0,
|
||||
"line_recall": 0.0,
|
||||
"line_f1": 0.0,
|
||||
"returned_lines": 0,
|
||||
"overlap_lines": 0
|
||||
},
|
||||
"status": "budget_exhausted",
|
||||
"elapsed_seconds": 16.60878831899754,
|
||||
"rounds": 6,
|
||||
"tool_calls": 5,
|
||||
"tool_errors": 1,
|
||||
"invalid_actions": 2,
|
||||
"input_tokens": 11097,
|
||||
"output_tokens": 191,
|
||||
"returned_chars": 0,
|
||||
"references": []
|
||||
},
|
||||
{
|
||||
"id": "requests-03",
|
||||
"repository": "requests",
|
||||
"scores": {
|
||||
"file_hit": true,
|
||||
"target_hit": false,
|
||||
"line_precision": 0.0,
|
||||
"line_recall": 0.0,
|
||||
"line_f1": 0.0,
|
||||
"returned_lines": 1,
|
||||
"overlap_lines": 0
|
||||
},
|
||||
"status": "completed",
|
||||
"elapsed_seconds": 11.590885819001414,
|
||||
"rounds": 4,
|
||||
"tool_calls": 4,
|
||||
"tool_errors": 0,
|
||||
"invalid_actions": 0,
|
||||
"input_tokens": 7341,
|
||||
"output_tokens": 134,
|
||||
"returned_chars": 79,
|
||||
"references": [
|
||||
{
|
||||
"path": "src/requests/sessions.py",
|
||||
"start_line": 284,
|
||||
"end_line": 284,
|
||||
"sha256": "0a5d5da449ce7f0af3ccf6e4bbe7a67a935e37846dff4ff9f08cb6c7e2464e6f",
|
||||
"verified": true
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "requests-04",
|
||||
"repository": "requests",
|
||||
"scores": {
|
||||
"file_hit": false,
|
||||
"target_hit": false,
|
||||
"line_precision": 0.0,
|
||||
"line_recall": 0.0,
|
||||
"line_f1": 0.0,
|
||||
"returned_lines": 0,
|
||||
"overlap_lines": 0
|
||||
},
|
||||
"status": "budget_exhausted",
|
||||
"elapsed_seconds": 19.878289842999948,
|
||||
"rounds": 6,
|
||||
"tool_calls": 5,
|
||||
"tool_errors": 2,
|
||||
"invalid_actions": 2,
|
||||
"input_tokens": 20986,
|
||||
"output_tokens": 210,
|
||||
"returned_chars": 0,
|
||||
"references": []
|
||||
},
|
||||
{
|
||||
"id": "requests-05",
|
||||
"repository": "requests",
|
||||
"scores": {
|
||||
"file_hit": false,
|
||||
"target_hit": false,
|
||||
"line_precision": 0.0,
|
||||
"line_recall": 0.0,
|
||||
"line_f1": 0.0,
|
||||
"returned_lines": 0,
|
||||
"overlap_lines": 0
|
||||
},
|
||||
"status": "budget_exhausted",
|
||||
"elapsed_seconds": 15.128609202998632,
|
||||
"rounds": 6,
|
||||
"tool_calls": 6,
|
||||
"tool_errors": 0,
|
||||
"invalid_actions": 1,
|
||||
"input_tokens": 11077,
|
||||
"output_tokens": 169,
|
||||
"returned_chars": 0,
|
||||
"references": []
|
||||
},
|
||||
{
|
||||
"id": "requests-06",
|
||||
"repository": "requests",
|
||||
"scores": {
|
||||
"file_hit": true,
|
||||
"target_hit": false,
|
||||
"line_precision": 0.0,
|
||||
"line_recall": 0.0,
|
||||
"line_f1": 0.0,
|
||||
"returned_lines": 65,
|
||||
"overlap_lines": 0
|
||||
},
|
||||
"status": "completed",
|
||||
"elapsed_seconds": 12.022663930001727,
|
||||
"rounds": 4,
|
||||
"tool_calls": 4,
|
||||
"tool_errors": 0,
|
||||
"invalid_actions": 0,
|
||||
"input_tokens": 12537,
|
||||
"output_tokens": 130,
|
||||
"returned_chars": 2038,
|
||||
"references": [
|
||||
{
|
||||
"path": "src/requests/auth.py",
|
||||
"start_line": 2,
|
||||
"end_line": 66,
|
||||
"sha256": "905ef9b6a9cb72d67d31ffe19bd4d9223e1c4169cde6ec51cfca16b31e70991d",
|
||||
"verified": true
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "requests-07",
|
||||
"repository": "requests",
|
||||
"scores": {
|
||||
"file_hit": false,
|
||||
"target_hit": false,
|
||||
"line_precision": 0.0,
|
||||
"line_recall": 0.0,
|
||||
"line_f1": 0.0,
|
||||
"returned_lines": 2,
|
||||
"overlap_lines": 0
|
||||
},
|
||||
"status": "completed",
|
||||
"elapsed_seconds": 16.638694857996597,
|
||||
"rounds": 6,
|
||||
"tool_calls": 5,
|
||||
"tool_errors": 1,
|
||||
"invalid_actions": 1,
|
||||
"input_tokens": 15935,
|
||||
"output_tokens": 185,
|
||||
"returned_chars": 71,
|
||||
"references": [
|
||||
{
|
||||
"path": "src/requests/compat.py",
|
||||
"start_line": 74,
|
||||
"end_line": 75,
|
||||
"sha256": "27bb088d1e97a031a9e494d5ccec642b97d2a145546bf3e373b8916610161a62",
|
||||
"verified": true
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "requests-08",
|
||||
"repository": "requests",
|
||||
"scores": {
|
||||
"file_hit": false,
|
||||
"target_hit": false,
|
||||
"line_precision": 0.0,
|
||||
"line_recall": 0.0,
|
||||
"line_f1": 0.0,
|
||||
"returned_lines": 0,
|
||||
"overlap_lines": 0
|
||||
},
|
||||
"status": "budget_exhausted",
|
||||
"elapsed_seconds": 17.901440784000442,
|
||||
"rounds": 6,
|
||||
"tool_calls": 5,
|
||||
"tool_errors": 3,
|
||||
"invalid_actions": 2,
|
||||
"input_tokens": 13474,
|
||||
"output_tokens": 207,
|
||||
"returned_chars": 0,
|
||||
"references": []
|
||||
},
|
||||
{
|
||||
"id": "requests-09",
|
||||
"repository": "requests",
|
||||
"scores": {
|
||||
"file_hit": false,
|
||||
"target_hit": false,
|
||||
"line_precision": 0.0,
|
||||
"line_recall": 0.0,
|
||||
"line_f1": 0.0,
|
||||
"returned_lines": 0,
|
||||
"overlap_lines": 0
|
||||
},
|
||||
"status": "budget_exhausted",
|
||||
"elapsed_seconds": 19.063683686996228,
|
||||
"rounds": 6,
|
||||
"tool_calls": 5,
|
||||
"tool_errors": 2,
|
||||
"invalid_actions": 2,
|
||||
"input_tokens": 16104,
|
||||
"output_tokens": 208,
|
||||
"returned_chars": 0,
|
||||
"references": []
|
||||
},
|
||||
{
|
||||
"id": "requests-10",
|
||||
"repository": "requests",
|
||||
"scores": {
|
||||
"file_hit": false,
|
||||
"target_hit": false,
|
||||
"line_precision": 0.0,
|
||||
"line_recall": 0.0,
|
||||
"line_f1": 0.0,
|
||||
"returned_lines": 16,
|
||||
"overlap_lines": 0
|
||||
},
|
||||
"status": "completed",
|
||||
"elapsed_seconds": 22.344112323000445,
|
||||
"rounds": 6,
|
||||
"tool_calls": 6,
|
||||
"tool_errors": 0,
|
||||
"invalid_actions": 0,
|
||||
"input_tokens": 21780,
|
||||
"output_tokens": 224,
|
||||
"returned_chars": 403,
|
||||
"references": [
|
||||
{
|
||||
"path": "src/requests/__init__.py",
|
||||
"start_line": 43,
|
||||
"end_line": 58,
|
||||
"sha256": "e3168011198f0c804fb1ad8fb23a54f6bd3aca8a0afb69992874d90215915adb",
|
||||
"verified": true
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "flask-01",
|
||||
"repository": "flask",
|
||||
"scores": {
|
||||
"file_hit": false,
|
||||
"target_hit": false,
|
||||
"line_precision": 0.0,
|
||||
"line_recall": 0.0,
|
||||
"line_f1": 0.0,
|
||||
"returned_lines": 1,
|
||||
"overlap_lines": 0
|
||||
},
|
||||
"status": "completed",
|
||||
"elapsed_seconds": 8.700928111000394,
|
||||
"rounds": 3,
|
||||
"tool_calls": 3,
|
||||
"tool_errors": 0,
|
||||
"invalid_actions": 0,
|
||||
"input_tokens": 7344,
|
||||
"output_tokens": 98,
|
||||
"returned_chars": 31,
|
||||
"references": [
|
||||
{
|
||||
"path": "src/flask/__init__.py",
|
||||
"start_line": 6,
|
||||
"end_line": 6,
|
||||
"sha256": "987bc937d4b0b65d510ed8c2a82218c889e22bf499fe5fe1a94ca73b382927da",
|
||||
"verified": true
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "flask-02",
|
||||
"repository": "flask",
|
||||
"scores": {
|
||||
"file_hit": false,
|
||||
"target_hit": false,
|
||||
"line_precision": 0.0,
|
||||
"line_recall": 0.0,
|
||||
"line_f1": 0.0,
|
||||
"returned_lines": 0,
|
||||
"overlap_lines": 0
|
||||
},
|
||||
"status": "budget_exhausted",
|
||||
"elapsed_seconds": 90.0734242529943,
|
||||
"rounds": 5,
|
||||
"tool_calls": 3,
|
||||
"tool_errors": 1,
|
||||
"invalid_actions": 3,
|
||||
"input_tokens": 10857,
|
||||
"output_tokens": 1099,
|
||||
"returned_chars": 0,
|
||||
"references": []
|
||||
},
|
||||
{
|
||||
"id": "flask-03",
|
||||
"repository": "flask",
|
||||
"scores": {
|
||||
"file_hit": true,
|
||||
"target_hit": false,
|
||||
"line_precision": 0.0,
|
||||
"line_recall": 0.0,
|
||||
"line_f1": 0.0,
|
||||
"returned_lines": 77,
|
||||
"overlap_lines": 0
|
||||
},
|
||||
"status": "completed",
|
||||
"elapsed_seconds": 10.396121550998942,
|
||||
"rounds": 4,
|
||||
"tool_calls": 4,
|
||||
"tool_errors": 0,
|
||||
"invalid_actions": 0,
|
||||
"input_tokens": 13402,
|
||||
"output_tokens": 104,
|
||||
"returned_chars": 2855,
|
||||
"references": [
|
||||
{
|
||||
"path": "src/flask/app.py",
|
||||
"start_line": 9,
|
||||
"end_line": 85,
|
||||
"sha256": "5c6aa0151b0b8018732280761d27eda0c4c83378630e21faadd57b73783720a7",
|
||||
"verified": true
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "flask-04",
|
||||
"repository": "flask",
|
||||
"scores": {
|
||||
"file_hit": false,
|
||||
"target_hit": false,
|
||||
"line_precision": 0.0,
|
||||
"line_recall": 0.0,
|
||||
"line_f1": 0.0,
|
||||
"returned_lines": 0,
|
||||
"overlap_lines": 0
|
||||
},
|
||||
"status": "budget_exhausted",
|
||||
"elapsed_seconds": 16.63429196999641,
|
||||
"rounds": 6,
|
||||
"tool_calls": 5,
|
||||
"tool_errors": 2,
|
||||
"invalid_actions": 2,
|
||||
"input_tokens": 16262,
|
||||
"output_tokens": 182,
|
||||
"returned_chars": 0,
|
||||
"references": []
|
||||
},
|
||||
{
|
||||
"id": "flask-05",
|
||||
"repository": "flask",
|
||||
"scores": {
|
||||
"file_hit": false,
|
||||
"target_hit": false,
|
||||
"line_precision": 0.0,
|
||||
"line_recall": 0.0,
|
||||
"line_f1": 0.0,
|
||||
"returned_lines": 0,
|
||||
"overlap_lines": 0
|
||||
},
|
||||
"status": "budget_exhausted",
|
||||
"elapsed_seconds": 15.005520599996089,
|
||||
"rounds": 6,
|
||||
"tool_calls": 1,
|
||||
"tool_errors": 0,
|
||||
"invalid_actions": 6,
|
||||
"input_tokens": 10690,
|
||||
"output_tokens": 175,
|
||||
"returned_chars": 0,
|
||||
"references": []
|
||||
},
|
||||
{
|
||||
"id": "flask-06",
|
||||
"repository": "flask",
|
||||
"scores": {
|
||||
"file_hit": false,
|
||||
"target_hit": false,
|
||||
"line_precision": 0.0,
|
||||
"line_recall": 0.0,
|
||||
"line_f1": 0.0,
|
||||
"returned_lines": 1,
|
||||
"overlap_lines": 0
|
||||
},
|
||||
"status": "completed",
|
||||
"elapsed_seconds": 10.608124224003404,
|
||||
"rounds": 4,
|
||||
"tool_calls": 4,
|
||||
"tool_errors": 0,
|
||||
"invalid_actions": 0,
|
||||
"input_tokens": 7948,
|
||||
"output_tokens": 119,
|
||||
"returned_chars": 66,
|
||||
"references": [
|
||||
{
|
||||
"path": "src/flask/app.py",
|
||||
"start_line": 1175,
|
||||
"end_line": 1175,
|
||||
"sha256": "5c6aa0151b0b8018732280761d27eda0c4c83378630e21faadd57b73783720a7",
|
||||
"verified": true
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "flask-07",
|
||||
"repository": "flask",
|
||||
"scores": {
|
||||
"file_hit": false,
|
||||
"target_hit": false,
|
||||
"line_precision": 0.0,
|
||||
"line_recall": 0.0,
|
||||
"line_f1": 0.0,
|
||||
"returned_lines": 0,
|
||||
"overlap_lines": 0
|
||||
},
|
||||
"status": "budget_exhausted",
|
||||
"elapsed_seconds": 15.543320686003426,
|
||||
"rounds": 6,
|
||||
"tool_calls": 1,
|
||||
"tool_errors": 0,
|
||||
"invalid_actions": 6,
|
||||
"input_tokens": 10692,
|
||||
"output_tokens": 170,
|
||||
"returned_chars": 0,
|
||||
"references": []
|
||||
},
|
||||
{
|
||||
"id": "flask-08",
|
||||
"repository": "flask",
|
||||
"scores": {
|
||||
"file_hit": false,
|
||||
"target_hit": false,
|
||||
"line_precision": 0.0,
|
||||
"line_recall": 0.0,
|
||||
"line_f1": 0.0,
|
||||
"returned_lines": 0,
|
||||
"overlap_lines": 0
|
||||
},
|
||||
"status": "budget_exhausted",
|
||||
"elapsed_seconds": 14.967171724005311,
|
||||
"rounds": 6,
|
||||
"tool_calls": 1,
|
||||
"tool_errors": 0,
|
||||
"invalid_actions": 6,
|
||||
"input_tokens": 10726,
|
||||
"output_tokens": 175,
|
||||
"returned_chars": 0,
|
||||
"references": []
|
||||
},
|
||||
{
|
||||
"id": "flask-09",
|
||||
"repository": "flask",
|
||||
"scores": {
|
||||
"file_hit": false,
|
||||
"target_hit": false,
|
||||
"line_precision": 0.0,
|
||||
"line_recall": 0.0,
|
||||
"line_f1": 0.0,
|
||||
"returned_lines": 15,
|
||||
"overlap_lines": 0
|
||||
},
|
||||
"status": "completed",
|
||||
"elapsed_seconds": 10.435718278997228,
|
||||
"rounds": 4,
|
||||
"tool_calls": 4,
|
||||
"tool_errors": 0,
|
||||
"invalid_actions": 0,
|
||||
"input_tokens": 8945,
|
||||
"output_tokens": 115,
|
||||
"returned_chars": 537,
|
||||
"references": [
|
||||
{
|
||||
"path": "src/flask/app.py",
|
||||
"start_line": 42,
|
||||
"end_line": 56,
|
||||
"sha256": "5c6aa0151b0b8018732280761d27eda0c4c83378630e21faadd57b73783720a7",
|
||||
"verified": true
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "flask-10",
|
||||
"repository": "flask",
|
||||
"scores": {
|
||||
"file_hit": false,
|
||||
"target_hit": false,
|
||||
"line_precision": 0.0,
|
||||
"line_recall": 0.0,
|
||||
"line_f1": 0.0,
|
||||
"returned_lines": 0,
|
||||
"overlap_lines": 0
|
||||
},
|
||||
"status": "budget_exhausted",
|
||||
"elapsed_seconds": 17.834809244995995,
|
||||
"rounds": 6,
|
||||
"tool_calls": 1,
|
||||
"tool_errors": 0,
|
||||
"invalid_actions": 6,
|
||||
"input_tokens": 10756,
|
||||
"output_tokens": 207,
|
||||
"returned_chars": 0,
|
||||
"references": []
|
||||
},
|
||||
{
|
||||
"id": "click-01",
|
||||
"repository": "click",
|
||||
"scores": {
|
||||
"file_hit": false,
|
||||
"target_hit": false,
|
||||
"line_precision": 0.0,
|
||||
"line_recall": 0.0,
|
||||
"line_f1": 0.0,
|
||||
"returned_lines": 0,
|
||||
"overlap_lines": 0
|
||||
},
|
||||
"status": "budget_exhausted",
|
||||
"elapsed_seconds": 55.50594035600079,
|
||||
"rounds": 6,
|
||||
"tool_calls": 3,
|
||||
"tool_errors": 1,
|
||||
"invalid_actions": 4,
|
||||
"input_tokens": 12969,
|
||||
"output_tokens": 661,
|
||||
"returned_chars": 0,
|
||||
"references": []
|
||||
},
|
||||
{
|
||||
"id": "click-02",
|
||||
"repository": "click",
|
||||
"scores": {
|
||||
"file_hit": false,
|
||||
"target_hit": false,
|
||||
"line_precision": 0.0,
|
||||
"line_recall": 0.0,
|
||||
"line_f1": 0.0,
|
||||
"returned_lines": 0,
|
||||
"overlap_lines": 0
|
||||
},
|
||||
"status": "budget_exhausted",
|
||||
"elapsed_seconds": 16.30554112500249,
|
||||
"rounds": 6,
|
||||
"tool_calls": 6,
|
||||
"tool_errors": 0,
|
||||
"invalid_actions": 1,
|
||||
"input_tokens": 10679,
|
||||
"output_tokens": 172,
|
||||
"returned_chars": 0,
|
||||
"references": []
|
||||
},
|
||||
{
|
||||
"id": "click-03",
|
||||
"repository": "click",
|
||||
"scores": {
|
||||
"file_hit": false,
|
||||
"target_hit": false,
|
||||
"line_precision": 0.0,
|
||||
"line_recall": 0.0,
|
||||
"line_f1": 0.0,
|
||||
"returned_lines": 3,
|
||||
"overlap_lines": 0
|
||||
},
|
||||
"status": "completed",
|
||||
"elapsed_seconds": 48.693233080004575,
|
||||
"rounds": 4,
|
||||
"tool_calls": 3,
|
||||
"tool_errors": 0,
|
||||
"invalid_actions": 1,
|
||||
"input_tokens": 8166,
|
||||
"output_tokens": 601,
|
||||
"returned_chars": 136,
|
||||
"references": [
|
||||
{
|
||||
"path": "src/click/parser.py",
|
||||
"start_line": 1,
|
||||
"end_line": 3,
|
||||
"sha256": "9d4d40876a75d6adbdba5d6f35d53db303e8fcf52e2abc54e985d3a22ef5ab57",
|
||||
"verified": true
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "click-04",
|
||||
"repository": "click",
|
||||
"scores": {
|
||||
"file_hit": false,
|
||||
"target_hit": false,
|
||||
"line_precision": 0.0,
|
||||
"line_recall": 0.0,
|
||||
"line_f1": 0.0,
|
||||
"returned_lines": 0,
|
||||
"overlap_lines": 0
|
||||
},
|
||||
"status": "budget_exhausted",
|
||||
"elapsed_seconds": 16.465862397999445,
|
||||
"rounds": 6,
|
||||
"tool_calls": 6,
|
||||
"tool_errors": 1,
|
||||
"invalid_actions": 1,
|
||||
"input_tokens": 10648,
|
||||
"output_tokens": 188,
|
||||
"returned_chars": 0,
|
||||
"references": []
|
||||
},
|
||||
{
|
||||
"id": "click-05",
|
||||
"repository": "click",
|
||||
"scores": {
|
||||
"file_hit": false,
|
||||
"target_hit": false,
|
||||
"line_precision": 0.0,
|
||||
"line_recall": 0.0,
|
||||
"line_f1": 0.0,
|
||||
"returned_lines": 0,
|
||||
"overlap_lines": 0
|
||||
},
|
||||
"status": "budget_exhausted",
|
||||
"elapsed_seconds": 18.45864745599829,
|
||||
"rounds": 6,
|
||||
"tool_calls": 6,
|
||||
"tool_errors": 0,
|
||||
"invalid_actions": 1,
|
||||
"input_tokens": 13371,
|
||||
"output_tokens": 206,
|
||||
"returned_chars": 0,
|
||||
"references": []
|
||||
},
|
||||
{
|
||||
"id": "click-06",
|
||||
"repository": "click",
|
||||
"scores": {
|
||||
"file_hit": true,
|
||||
"target_hit": false,
|
||||
"line_precision": 0.0,
|
||||
"line_recall": 0.0,
|
||||
"line_f1": 0.0,
|
||||
"returned_lines": 1,
|
||||
"overlap_lines": 0
|
||||
},
|
||||
"status": "completed",
|
||||
"elapsed_seconds": 11.089595678000478,
|
||||
"rounds": 4,
|
||||
"tool_calls": 4,
|
||||
"tool_errors": 0,
|
||||
"invalid_actions": 0,
|
||||
"input_tokens": 9696,
|
||||
"output_tokens": 120,
|
||||
"returned_chars": 72,
|
||||
"references": [
|
||||
{
|
||||
"path": "src/click/core.py",
|
||||
"start_line": 100,
|
||||
"end_line": 100,
|
||||
"sha256": "814869352f5c14119d1175dd8ac1951be791bc67f8f514f215a831ba5ab0745c",
|
||||
"verified": true
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "click-07",
|
||||
"repository": "click",
|
||||
"scores": {
|
||||
"file_hit": false,
|
||||
"target_hit": false,
|
||||
"line_precision": 0.0,
|
||||
"line_recall": 0.0,
|
||||
"line_f1": 0.0,
|
||||
"returned_lines": 6,
|
||||
"overlap_lines": 0
|
||||
},
|
||||
"status": "completed",
|
||||
"elapsed_seconds": 11.229277147001994,
|
||||
"rounds": 4,
|
||||
"tool_calls": 4,
|
||||
"tool_errors": 0,
|
||||
"invalid_actions": 0,
|
||||
"input_tokens": 9244,
|
||||
"output_tokens": 118,
|
||||
"returned_chars": 217,
|
||||
"references": [
|
||||
{
|
||||
"path": "src/click/__init__.py",
|
||||
"start_line": 52,
|
||||
"end_line": 57,
|
||||
"sha256": "e98c92d5a7b29276742d8c1e5a8ccd672d00f6767fd75c2660886fddc6d0ad8a",
|
||||
"verified": true
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "click-08",
|
||||
"repository": "click",
|
||||
"scores": {
|
||||
"file_hit": false,
|
||||
"target_hit": false,
|
||||
"line_precision": 0.0,
|
||||
"line_recall": 0.0,
|
||||
"line_f1": 0.0,
|
||||
"returned_lines": 4,
|
||||
"overlap_lines": 0
|
||||
},
|
||||
"status": "completed",
|
||||
"elapsed_seconds": 12.85814527599723,
|
||||
"rounds": 5,
|
||||
"tool_calls": 5,
|
||||
"tool_errors": 0,
|
||||
"invalid_actions": 0,
|
||||
"input_tokens": 9711,
|
||||
"output_tokens": 140,
|
||||
"returned_chars": 211,
|
||||
"references": [
|
||||
{
|
||||
"path": "src/click/decorators.py",
|
||||
"start_line": 381,
|
||||
"end_line": 384,
|
||||
"sha256": "e4feda6e126d010629fca1e08d4be132fe3ae044703b3aefd9e9cd9279701f24",
|
||||
"verified": true
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "click-09",
|
||||
"repository": "click",
|
||||
"scores": {
|
||||
"file_hit": false,
|
||||
"target_hit": false,
|
||||
"line_precision": 0.0,
|
||||
"line_recall": 0.0,
|
||||
"line_f1": 0.0,
|
||||
"returned_lines": 1,
|
||||
"overlap_lines": 0
|
||||
},
|
||||
"status": "completed",
|
||||
"elapsed_seconds": 10.134168666998448,
|
||||
"rounds": 4,
|
||||
"tool_calls": 4,
|
||||
"tool_errors": 0,
|
||||
"invalid_actions": 0,
|
||||
"input_tokens": 7359,
|
||||
"output_tokens": 116,
|
||||
"returned_chars": 27,
|
||||
"references": [
|
||||
{
|
||||
"path": "src/click/termui.py",
|
||||
"start_line": 806,
|
||||
"end_line": 806,
|
||||
"sha256": "bc062b282d9aedffcd7c42210138445587dafff89be5741b4d2086b499400510",
|
||||
"verified": true
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "click-10",
|
||||
"repository": "click",
|
||||
"scores": {
|
||||
"file_hit": false,
|
||||
"target_hit": false,
|
||||
"line_precision": 0.0,
|
||||
"line_recall": 0.0,
|
||||
"line_f1": 0.0,
|
||||
"returned_lines": 0,
|
||||
"overlap_lines": 0
|
||||
},
|
||||
"status": "budget_exhausted",
|
||||
"elapsed_seconds": 53.11997275899921,
|
||||
"rounds": 6,
|
||||
"tool_calls": 3,
|
||||
"tool_errors": 1,
|
||||
"invalid_actions": 4,
|
||||
"input_tokens": 12999,
|
||||
"output_tokens": 661,
|
||||
"returned_chars": 0,
|
||||
"references": []
|
||||
}
|
||||
]
|
||||
@@ -0,0 +1,34 @@
|
||||
{
|
||||
"kind": "executed_oracle_demonstrations",
|
||||
"seed": 42,
|
||||
"limitations": "Three-file synthetic repositories; query-word searches and oracle-chosen target files and final ranges. Teaches protocol, not realistic planning.",
|
||||
"recipe": "read-windows",
|
||||
"read_window": {
|
||||
"lines_before_match": 40,
|
||||
"lines_after_match": 60
|
||||
},
|
||||
"functions_per_file": 2,
|
||||
"target_position": "randomized before or after a distractor function",
|
||||
"excluded_repositories_by_name": [
|
||||
"click",
|
||||
"flask",
|
||||
"requests"
|
||||
],
|
||||
"splits": {
|
||||
"train": {
|
||||
"trajectories": 256,
|
||||
"action_examples": 826,
|
||||
"unobservable_candidates_skipped": 1,
|
||||
"source_sha256": "3eeed3185b74f934462aaccbd3c567f8db20754a1bf6f5f3686779f8b39f4338",
|
||||
"output_sha256": "7c6315531aa37e5de59c6944a425cb9cb700474551a62a3fc51dd58134649b7b"
|
||||
},
|
||||
"validation": {
|
||||
"trajectories": 32,
|
||||
"action_examples": 104,
|
||||
"unobservable_candidates_skipped": 0,
|
||||
"source_sha256": "e1b934c33f12322e4a56d0d17a987966c40961b035e15939fd716b76d4e017a2",
|
||||
"output_sha256": "ac270e077c3e65f5590c18f27a3dec1fd80c48aac6067bc81b0e05f9b2cb0bc1"
|
||||
}
|
||||
},
|
||||
"repository_disjoint": true
|
||||
}
|
||||
Reference in New Issue
Block a user