Processed continuous-waveform database for reproducible regional monitoring studies
## 1. Overview **SeismicX-Cont** is a processed and structured HDF5-based continuous-waveform data product for deployment-level earthquake-monitoring studies. The release preserves continuous waveform context, station-time coverage, annotation provenance, instrument-response metadata, and validation lineage in a common file organization. It is intended as a reusable monitoring data product, not as a relative model-assessment resource or a replacement for the authoritative SCEDC, NCEDC, or CEED source archives. The repository includes the **SeismicX DataLoader**, SQLite coverage-query utilities, validation scripts, and quick-start notebooks for reproducible access to the released files. ## Dataset preview The Hugging Face Dataset Viewer displays a lightweight file index rather than the full HDF5 waveform contents. The waveform data are stored as structured HDF5 files and should be accessed using the provided documentation and scripts. The preview table is intended only to summarize available files, time periods, networks, and data types. --- ## 1.1 Repository Layout The repository is organized as follows: ```text data/ hdf5/ # 14 daily HDF5 waveform files index/ # SQLite waveform index label/ # annotations, reference tables, optional consensus and multi-output catalogue response/ # instrument-response JSON for dataloader correction/simulation validation/ # machine-readable audits and optional phase-matching diagnostics notebooks/ # interactive quickstart notebook pickers/ # example TorchScript/ONNX picker models scripts/ # conversion, indexing, picker, association, and upload scripts essd_scripts/ # ESSD statistics, diagnostics, and plotting scripts utils/ # reusable HDF5 dataloader and waveform-index APIs ``` The main data-preparation and picker entry points live in `scripts/`; ESSD manuscript numbers, statistics, diagnostics, and plotting entry points live in `essd_scripts/`. Use `essd_scripts/audit_manuscript_numbers.py` to regenerate inventory and exact NSLC point-coverage values, including finite-sample checks for floating-point HDF5 arrays, independently of all picker outputs. Use `essd_scripts/build_reference_arrivals.py` to regenerate C0 point coverage and configuration-specific C1--C3 window, component, gap, sampling-rate, and response fields in `data/label/reference_arrivals.sqlite`. The database preserves all source readings in `reference_arrivals`, stores one canonical station-location-phase-time denominator row in `evaluation_references`, and links every canonical row back to its source arrivals and event identifiers through `evaluation_reference_members`. The released configuration selects one location and channel family deterministically, never combines components across locations or families, and uses half-open response epochs. Response-qualified C3 requires exactly one matching response epoch for every selected component. Use `essd_scripts/audit_waveform_quality.py` for the full index-integrity audit and the deterministic period/network/channel-family-stratified HDF5 sample reported in the manuscript. Canonical machine-readable summaries are released as `data/validation/manuscript_number_audit.json` and `data/validation/waveform_quality_audit.json`; the reproduction script refreshes them from the distributed data files. The optional `data/validation/multimodel_phase_matching/` extension stores automatically regenerated one-to-one phase-agreement records and summaries for non-standardized example outputs. Regenerate it with `./essd_scripts/regenerate_multimodel_phase_matching.sh`; it is an automated consistency and interoperability check, not a model comparison or ground truth. The script reads versioned C0 flags from `reference_arrivals.sqlite`, which was independently generated from exact NSLC intervals and finite HDF5 samples. The optional compressed multi-output catalogue under `data/label/` retains every source pick from nine non-standardized example runs and groups station-phase records with a strict span below 1.5 s. It provides compact JSONL and normalized SQLite representations for querying output agreement. The adjacent `output_eligibility.sqlite` records processed station-time intervals, and `output_registry.tsv` documents the nine source output streams. The released support-stratified validation tables report candidate-centred agreement and manual-reference recovery, both before and after normalization by the number of eligible outputs. These summaries support the two explicitly conditioned phase-pick precision proxies described in the manuscript; they do not claim an exhaustive conventional full-stream precision measurement. Output support is not ground truth or a relative model assessment. Workflow-level results are released in `data/validation/leave_one_output_precision_proxy.tsv`, including candidate reconstruction after removing each assessed output and a 20-replicate station-day time-shift null for the declared example output. The catalogue-conditioned window sensitivity is released in `data/validation/reference_window_precision_proxy_sensitivity.tsv` for `T=1,5,10,30` s. A collection of serialized picker artifacts used by the project is provided under `pickers/`. The PNSN implementation and related phase-detection models are additionally archived at https://doi.org/10.5281/zenodo.17510174, and the PNSN workflow is described at https://doi.org/10.1029/2025JH000944. See `pickers/README.md` for the output-to-artifact mapping and `data/label/README_all_model_phase_picks.md` for the candidate schema and query examples. The quickstart notebook `notebooks/seismicx_cont_quickstart.ipynb` demonstrates file checks, SQLite queries, HDF5 waveform plotting, annotation inspection, and evaluation command templates. Current `scripts/` contents: | Path | Purpose | | --- | --- | | `scripts/README_download_data.md` | Download and rebuild notes for full and mini data products. | | `scripts/README_event_comparison.md` | Event-comparison workflow notes and metric definitions. | | `scripts/README_real_association.md` | REAL association setup, inputs, outputs, and tuning notes. | | `scripts/build_consensus_picks_json.py` | Build consensus picker annotations from model pick outputs. | | `scripts/build_all_model_phase_catalog.py` | Build the compact multi-output phase-candidate JSONL and SQLite products. | | `scripts/build_output_eligibility.py` | Build processed station-time eligibility intervals and the output registry. | | `scripts/build_waveform_index.sh` | Build the full-release SQLite waveform index from HDF5 files. | | `scripts/compare_associated_events.py` | Compare associated events against catalog/reference events. | | `scripts/download_instrument_responses.py` | Download station instrument-response metadata. | | `scripts/evaluate_picks.py` | Evaluate picker JSONL outputs against reference picks. | | `scripts/example_dataloader.py` | Minimal PyTorch `HDF5WaveformDataset` usage example. | | `scripts/example_query_waveform.py` | Query waveform segments from SQLite and optionally process response. | | `scripts/getwaveform_cha_names.py` | Inspect channel-family names available in HDF5 waveform files. | | `scripts/makeh5.py` | Convert MiniSEED waveform files into the original HDF5 layout. | | `scripts/makeh5_flex_seg.py` | Convert waveforms into flexible-segment HDF5 files. | | `scripts/plot_dataset_overview.py` | Generate dataset-overview plots. | | `scripts/plot_essd_qc_figures.py` | Generate ESSD quality-control figures. | | `scripts/readh5.py` | Inspect any HDF5 hierarchy, datasets, links, and attributes. | | `scripts/run_consensus_picks.sh` | Run the full-release consensus-pick workflow. | | `scripts/run_all_model_phase_catalog.sh` | Run the resumable nine-output candidate-catalogue build. | | `scripts/run_picker_to_jsonl.py` | Run one TorchScript picker over HDF5 samples and write JSONL picks. | | `scripts/run_pickers.sh` | Batch-run full-release picker models. | | `scripts/run_real_association.py` | Convert picker JSONL to REAL inputs and associate events. | | `scripts/upload_files_to_hf.py` | Upload selected full-release data files to Hugging Face. | | `scripts/upload_files_to_ms.py` | Upload selected full-release data files to ModelScope. | | `scripts/upload_restfiles_hf.py` | Upload non-bulk release files to Hugging Face. | | `scripts/upload_restfiles_ms.py` | Upload `scripts/` and `utils/` updates to ModelScope. | | `scripts/upload_all_model_phase_catalog.py` | Upload the compressed multi-output catalogue and its documentation. | | `scripts/validate_multi_output_candidates.py` | Reproduce support-stratified C0 manual-reference agreement. | | `scripts/validate_leave_one_output_precision_proxy.py` | Reconstruct external candidates and calculate workflow-level precision proxy 2 plus the station-day null. | | `scripts/validate_reference_window_precision_proxy.py` | Calculate precision proxy 1 and its reference-window sensitivity. | | `scripts/verify_full_dataset.py` | Verify full-release required files and basic structure. | | `scripts/write_manifest.sh` | Regenerate release checksum and file-size manifests. | Release metadata and integrity files: - `VERSION` records the release version (`v1.0.0`). - `CITATION.cff` provides machine-readable citation metadata. - `LICENSE_CLARIFICATION.md` explains the layer-specific data-use terms. - `manifest_sha256.txt` lists release-file SHA-256 checksums. - `manifest_filesizes.tsv` lists release-file sizes in bytes. - Full release: https://huggingface.co/datasets/cangyeone/SeismicX-Cont. The version-specific DOI for the frozen submission release is cited in the ESSD manuscript and on the Hugging Face DOI panel. - Mini two-hour dataset DOI (Zenodo): https://doi.org/10.5281/zenodo.21472154 - Mini two-hour dataset DOI (Hugging Face): https://doi.org/10.57967/hf/9664 (Revision `883e092`) - Public mirror: https://www.modelscope.cn/datasets/cangyeone/SeismicX-Cont Recommended quickstart and reproducibility order: 1. Inspect the repository layout and file checks with `notebooks/seismicx_cont_quickstart.ipynb`. 2. Verify the full release structure with `python scripts/verify_full_dataset.py`. 3. Build or refresh the SQLite waveform index with `./scripts/build_waveform_index.sh` when rebuilding from HDF5 files. 4. Regenerate the release checksum and file-size manifests with `./scripts/write_manifest.sh`. 5. Reproduce ESSD manuscript tables and figures with `./essd_scripts/reproduce_manuscript_outputs.sh`. 6. If the optional source picker JSONL files are present, regenerate the multi-output agreement layer with `./essd_scripts/regenerate_multimodel_phase_matching.sh`. 7. Query the optional multi-output candidate catalogue after decompressing `data/label/all_model_phase_picks.sqlite.zst`; schema and examples are in `data/label/README_all_model_phase_picks.md`. 8. Use `data/label/output_eligibility.sqlite` for processed-domain queries and inspect `data/validation/multi_output_manual_reference_match.tsv` and `data/validation/multi_output_eligibility_normalized_match.tsv` for the released support-stratified catalogue-agreement summaries. 9. Inspect `data/validation/leave_one_output_precision_proxy.tsv` and `data/validation/reference_window_precision_proxy_sensitivity.tsv` for the workflow-level and reference-window precision-proxy results. A tested environment is macOS 26.5.2 on Apple silicon with Python 3.12.2. Exact versions of the principal runtime packages are listed in `requirements.txt`. Install them in an isolated environment with `python -m pip install -r requirements.txt`. The released scripts use only the Python standard library, the listed third-party packages, and modules distributed under `scripts/`, `essd_scripts/`, and `utils/`; no unpublished local Python module is required for the documented access and validation paths. --- ## 2. Dataset Description ### 2.1 Data Source This dataset contains continuous waveform records from three regional seismic networks: - CI — Southern California Seismic Network - NC — Northern California Seismic Network - BK — Berkeley Digital Seismic Network These networks present varied observation conditions, including differences in instrument response, channel configuration, noise characteristics, and station density. --- ### 2.2 Time Windows Two representative 7-day continuous time windows are provided: ```python TIME_WINDOWS = [ { "id": "day_20190701", "starttime": "2019-07-01T00:00:00", "endtime": "2019-07-08T00:00:00", }, { "id": "day_20211108", "starttime": "2021-11-08T00:00:00", "endtime": "2021-11-15T00:00:00", }, ] ``` Each window covers **7 consecutive days of continuous seismic recordings**. --- ### 2.3 Data Content The dataset includes continuous three-component waveform data in HDF5 format, station metadata (latitude, longitude, elevation, validity period), multi-segment waveform records merged into continuous traces, and event annotations derived from the **CEED dataset**. --- ## 3. Monitoring-Period Design SeismicX-Cont pairs two one-week monitoring periods in the same HDF5, annotation, response, and coverage-index structure. The paired design supports reusable continuous-stream validation under contrasting monitoring conditions. ### 3.1 High-Activity Window (July 2019) Dense seismicity with frequent and overlapping events. Reuse typically focuses on coverage-qualified recall, phase-time residuals, station coverage, and the ability to inspect workflow behavior during clustered earthquake sequences. ### 3.2 Low-Activity Window (November 2021) Sparse seismicity with extended quiet periods relative to the Ridgecrest week. Reuse typically focuses on catalog-unmatched automatic-pick burden, noise conditions, station availability, and reporting stability over long continuous intervals. Catalog-unmatched picks should not be interpreted automatically as confirmed false detections. --- ## 4. Data Structure The dataset is stored in a hierarchical HDF5 layout: ``` /year /day /stations /network.station.location /waveform /channel /segment ``` Example: ``` /2019-01-01T00:00:00.000000Z /2019-07-01T00:00:00.000000Z /stations /BK.BDM.00 /waveform /BHE/0 /BHN/0 /BHZ/0 ``` Each channel can contain multiple waveform segments that are automatically merged during loading. --- ## 5. SeismicX DataLoader **File:** `utils/hdf5_waveform_dataset.py` A PyTorch `Dataset` that wraps the hierarchical HDF5 files and delivers ready-to-use waveform tensors with full station metadata. Designed for single-pass inference over large multi-file datasets with minimal memory footprint. --- ### 5.1 Key Features **Multi-file input** — accepts a single file, a directory, a glob pattern (`data/hdf5/*.h5`), or a Python list of paths. **Channel-family grouping** — waveforms are grouped by the first two characters of the channel code (the "family"), so BHE/BHN/BHZ form one sample rather than three separate ones. | Channels | Family | | --- | --- | | BHE, BHN, BHZ | BH | | HHE, HHN, HHZ | HH | | HNE, HNN, HNZ / HN1, HN2, HN3 | HN | | EHE, EHN, EHZ | EH | **Automatic 3-component construction** — E/1 → slot 0, N/2 → slot 1, Z/3 → slot 2. Z-only stations can be replicated to `[Z, Z, Z]`. **Multi-segment merging** — consecutive HDF5 segments are merged into one contiguous trace with gap-filling. **Optional resampling** — linear-interpolation resampling to a target sampling rate (no scipy dependency). **Optional instrument-response handling** — the loader can remove the native instrument response using `data/response/instrument_responses.json` and can optionally simulate a user-selected target response. **Memory-safe file handle caching** — only the current HDF5 file's handle is kept open; when the loader moves to a new file the old handle (and its chunk cache) is closed immediately. Memory usage stays O(1) regardless of the number of files. --- ### 5.2 Constructor Parameters ```python HDF5WaveformDataset( h5_file, mode="three", fill_value=0.0, dtype=np.float32, default_location="--", allowed_families=("HH", "BH", "EH", "HN"), allowed_z_only_channels=("EHZ",), allow_z_only=True, replicate_z_only=False, target_sampling_rate=None, skip_sample_keys=None, skip_jsonl=None, skip_record_type="phase_pick", keep_h5_open=True, include_segments_metadata=True, use_overlap_mask=True, h5_rdcc_nbytes=8 * 1024 * 1024, max_duration_sec=90000.0, instrument_response_json=None, remove_instrument_response=False, response_output="VEL", response_pre_filt=None, response_water_level=60, response_error_behavior="raise", simulate_instrument_response=False, simulation_response_json=None, simulation_response_id=None, simulation_response_selector=None, simulation_paz=None, simulation_output=None, ) ``` | Parameter | Type | Default | Description | | --- | --- | --- | --- | | `h5_file` | str / Path / list | required | Single file, directory, glob pattern (`data/hdf5/*.h5`), or list | | `mode` | str | `"three"` | Output mode: `"single"`, `"three"`, or `"multi"` | | `allowed_families` | tuple[str] | `("HH","BH","EH","HN")` | Channel families included in the index | | `allowed_z_only_channels` | tuple[str] | `("EHZ",)` | Full channel codes accepted as Z-only stations | | `allow_z_only` | bool | `True` | Include single-Z stations (from `allowed_z_only_channels`) | | `replicate_z_only` | bool | `False` | Opt in to synthetic [Z,Z,Z] input for a Z-only record | | `target_sampling_rate` | float or None | `None` | Resample to this rate (Hz); `None` keeps original rate | | `fill_value` | float | `0.0` | Value used to fill gaps between merged segments | | `dtype` | np.dtype | `np.float32` | NumPy dtype for waveform arrays | | `keep_h5_open` | bool | `True` | Cache the current file's handle; stale handles are closed | | `include_segments_metadata` | bool | `True` | Include raw segment dicts in the output dict (set `False` to save memory) | | `use_overlap_mask` | bool | `True` | Prevent overlapping segments from overwriting each other | | `h5_rdcc_nbytes` | int | `8 MB` | HDF5 chunk-cache size per file handle (8 MB is enough for single-pass inference) | | `max_duration_sec` | float | `90000.0` | Safety limit for one merged channel trace | | `instrument_response_json` | str or None | `None` | Path to `data/response/instrument_responses.json` for response removal | | `remove_instrument_response` | bool | `False` | Remove native instrument response with ObsPy and return physical units | | `response_output` | str | `"VEL"` | Output unit for response removal: usually `"DISP"`, `"VEL"`, or `"ACC"` | | `response_pre_filt` | tuple or None | `None` | Four-corner frequency pre-filter passed to ObsPy `remove_response` | | `response_water_level` | float or None | `60` | Water level passed to ObsPy; use `None` when controlling stabilization by `response_pre_filt` | | `response_error_behavior` | str | `"raise"` | `"raise"`, `"warn"`, or `"skip"` if a response cannot be applied | | `simulate_instrument_response` | bool | `False` | Simulate a target response after optional response removal | | `simulation_response_json` | str or None | `None` | Response JSON containing the target simulated response; defaults to `instrument_response_json` | | `simulation_response_id` | str or None | `None` | Exact `response_id` to simulate | | `simulation_response_selector` | dict or None | `None` | Select target response by `network`, `station`, `location`, `channel`, and optional `time` | | `simulation_paz` | dict or None | `None` | Alternative ObsPy-style PAZ dict for simple response simulation | | `simulation_output` | str or None | `None` | Physical unit used when evaluating the target response; defaults to `response_output` | | `skip_sample_keys` | set or None | `None` | Pre-built set of sample keys to exclude from the index | | `skip_jsonl` | str or None | `None` | Path to an existing output JSONL; finished samples are scanned and removed from the index at startup | | `skip_record_type` | str | `"phase_pick"` | `record_type` value matched when scanning the JSONL for resume | --- ### 5.3 Output Modes #### `mode="single"` One sample per channel. Output `waveform` shape: `[T]`. #### `mode="three"` (recommended) One sample per channel family. Observed components are arranged as E/N/Z (or 1/2/3 mapped to the same slots). A Z-only record has shape `[T, 1]` by default; synthetic `[Z,Z,Z]` input requires `replicate_z_only=True`. #### `mode="multi"` One sample per channel family, all available channels stacked. Output `waveform` shape: `[T, C]`. Every mode returns `validity_mask`, which is false for gap-filled samples, and `component_mask`, which identifies observed output components. Three-component mode also returns `observed_component_mask`; replicated Z traces never become observed horizontal components in this mask. Resampling uses an anti-aliased polyphase filter when SciPy is available, and the validity mask is resampled conservatively so interpolation does not hide gaps. --- ### 5.4 Usage Examples **Single file:** ```python from utils.hdf5_waveform_dataset import HDF5WaveformDataset, waveform_collate_fn dataset = HDF5WaveformDataset( h5_file="data/hdf5/continuous_waveform_usa_20190701.h5", mode="three", target_sampling_rate=100.0, ) ``` **Glob pattern (multiple files):** ```python dataset = HDF5WaveformDataset( h5_file="data/hdf5/continuous_waveform_usa_*.h5", mode="three", allowed_families=("HH", "BH", "EH", "HN"), allow_z_only=True, replicate_z_only=True, target_sampling_rate=100.0, ) print(f"Files: {len(dataset.h5_files)} | Samples: {len(dataset)}") ``` **Directory:** ```python dataset = HDF5WaveformDataset(h5_file="data/hdf5/", mode="three") ``` **Remove native instrument response:** ```python dataset = HDF5WaveformDataset( h5_file="data/hdf5/continuous_waveform_usa_20190701.h5", mode="three", instrument_response_json="data/response/instrument_responses.json", remove_instrument_response=True, response_output="VEL", response_pre_filt=(0.2, 0.5, 20.0, 45.0), response_water_level=None, ) ``` **Remove the native response and simulate a target response:** ```python dataset = HDF5WaveformDataset( h5_file="data/hdf5/continuous_waveform_usa_20190701.h5", mode="single", instrument_response_json="data/response/instrument_responses.json", remove_instrument_response=True, response_output="VEL", response_pre_filt=(0.2, 0.5, 20.0, 45.0), response_water_level=None, simulate_instrument_response=True, simulation_response_selector={ "network": "BK", "station": "BDM", "location": "00", "channel": "BHZ", "time": "2019-07-06T04:00:00Z", }, simulation_output="VEL", ) ``` The returned item includes an `instrument_processing` field with the matched native `response_id`, response epoch, simulated response id, and any processing error. Response removal and response simulation use ObsPy's response machinery; the dataloader only provides the SeismicX-Cont JSON lookup and per-sample matching layer. --- ### 5.5 DataLoader Configuration **Single-process (safest — required for MPS):** ```python from torch.utils.data import DataLoader loader = DataLoader( dataset, batch_size=1, shuffle=False, num_workers=0, collate_fn=waveform_collate_fn, ) ``` **Multi-process (recommended for CUDA):** h5py is not fork-safe. On Linux the default DataLoader start method is `fork`, which can corrupt HDF5 reads. Use `spawn` explicitly: ```python from utils.hdf5_waveform_dataset import hdf5_worker_init_fn loader = DataLoader( dataset, batch_size=4, shuffle=False, num_workers=4, collate_fn=waveform_collate_fn, multiprocessing_context="spawn", # avoids h5py + fork deadlocks on Linux worker_init_fn=hdf5_worker_init_fn, prefetch_factor=2, persistent_workers=True, pin_memory=True, # speeds up CPU → GPU transfers on CUDA ) ``` On macOS the default context is already `spawn` (Python ≥ 3.8), so no override is needed. --- ### 5.6 Resume: Skip Already-Processed Samples Pass `skip_jsonl` to remove already-processed samples from the index **before any waveform data is read from HDF5**: ```python dataset = HDF5WaveformDataset( h5_file="data/hdf5/continuous_waveform_usa_*.h5", mode="three", skip_jsonl="data/picks/output.jsonl", skip_record_type="phase_pick", ) print(f"Original samples : {dataset.original_index_size}") print(f"Already done : {dataset.filtered_index_size}") print(f"Remaining : {len(dataset)}") ``` At startup the JSONL is split into byte-range chunks and parsed in parallel with `ThreadPoolExecutor`. Each finished record's sample key is reconstructed directly from its fields (`h5_file`, `year_id`, `day_id`, `station_id`, `channel_family`, `channels`). The matching index entries are dropped before any `__getitem__` call. --- ### 5.7 Output Sample Dictionary Each item returned by `__getitem__` (or delivered by the DataLoader after `waveform_collate_fn`) is a plain Python `dict`: ```python { # Identity "mode": "three", "h5_file": "data/hdf5/continuous_waveform_usa_20190701.h5", "year_id": "2019-01-01T00:00:00.000000Z", "day_id": "2019-07-01T00:00:00.000000Z", "station_id": "BK.BDM.00", # Channel layout "channel_family": "BH", "channels": ["BHE", "BHN", "BHZ"], # one entry per component slot "component_order": "E/N/Z or 1/2/3", "is_z_only": False, "z_only_replicated": False, # Waveform "waveform": torch.Tensor, # shape [T, 3], dtype float32 "sampling_rate": 100.0, # Hz after optional resampling "original_sampling_rate": 40.0, "target_sampling_rate": 100.0, "resampled": True, "npts": 8640001, # Timing "starttime": "2019-07-01T00:00:00.000000Z", "endtime": "2019-07-01T23:59:59.990000Z", # Station metadata "station_info": { "station_id": "BK.BDM.00", "network": "BK", "station": "BDM", "location": "00", "longitude": -122.2486, "latitude": 37.8762, "elevation": 276.0, "location_available": True, "position_match_mode": "strict_time_matched_network_station_only", "position_is_fallback": False, "position_history": [...], ... }, # Per-channel metadata (keyed by channel code) "channel_info": { "BHE": {"channel": "BHE", "segment_count": 3, "starttime": "...", ...}, "BHN": {...}, "BHZ": {...}, }, # Raw segment metadata (empty list when include_segments_metadata=False) "segments": [], } ``` --- ### 5.8 Minimal Example Script `scripts/example_dataloader.py` demonstrates the full pipeline in under 90 lines: ```bash python scripts/example_dataloader.py \ --h5_input data/hdf5/continuous_waveform_usa_20190701.h5 \ --n_samples 3 ``` Sample output: ``` ── Sample 1 ────────────────────────────────────────── station_id : BK.BDM.00 network : BK.BDM channels : ['BHE', 'BHN', 'BHZ'] starttime : 2019-07-01T00:00:00.000000Z sampling_rate : 100.0 Hz waveform shape: (8640001, 3) (86400.0 s × 3 components) waveform dtype: torch.float32 Z-only : False location : lon=-122.2486 lat=37.8762 ch[E/1] min=-1.234e-05 max=+2.345e-05 std=3.210e-06 ch[N/2] min=-1.100e-05 max=+1.987e-05 std=2.890e-06 ch[Z/3] min=-9.870e-06 max=+1.543e-05 std=2.654e-06 ``` --- ## 6. Recommended Reporting Quantities Recommended reporting quantities for reusable continuous-monitoring studies: - Declared coverage level (C0--C3), NSLC families, window, component, sampling, response, and gap requirements. - Coverage-aware recall: the label-centred fraction of coverage-qualified reference arrivals matched within the declared tolerance. - Pick-centred matched fraction: the fraction of eligible automatic picks used by the one-to-one assignment within that tolerance. - Total automatic-pick volume and catalogue-unmatched pick volume per unit time; unmatched picks are not presumed to be false detections. - Thresholds, preprocessing, channel selection, and phase normalization. - Residual summaries when needed for timing diagnostics. --- ## 7. Data-Product Rationale SeismicX-Cont is organized as a reusable continuous-monitoring data product with separate reporting views for: - high-event-density conditions, where waveform coverage and reference-label denominators must be documented carefully; - lower-event-rate background conditions, where catalog-unmatched automatic picks, station coverage, and metadata completeness become prominent. The structure supports reproducible reporting and workflow diagnostics without implying a standardized or fair comparison among independently trained picking systems. --- ## 8. Summary SeismicX-Cont provides a processed continuous waveform database, coverage metadata, CEED-derived annotations, instrument-response metadata, validation scripts, and reusable access utilities. It is well suited for continuous seismic detection studies, monitoring-workflow diagnostics, phase-association experiments, and large-scale waveform analysis where provenance, coverage, and source-provider attribution must remain auditable. --- ## 9. Waveform Index API and Data Query (with macOS External Drive Notes) This section introduces the SQLite-based waveform indexing and querying system. ### 9.1 Core API ```python from utils.waveform_index_api import query_and_merge merged, rows = query_and_merge( db_file="data/index/waveform_index.sqlite", network="BK", station="BDM", location="*", channel="BH*", starttime="2019-07-01T00:00:00", endtime="2019-07-01T01:00:00", fill_value=0.0, ) ``` `merged` is a dict keyed by `"network.station.location.channel"`, each value containing `data` (ndarray), `sampling_rate`, `starttime`, `endtime`, `filled_ratio`, and `segments`. The same API can apply the released instrument-response JSON before returning arrays: ```python merged, rows = query_and_merge( db_file="data/index/waveform_index.sqlite", network="BK", station="BDM", channel="BHZ", starttime="2019-07-01T00:00:00", endtime="2019-07-01T00:01:00", instrument_response_json="data/response/instrument_responses.json", remove_instrument_response=True, response_output="VEL", response_pre_filt=(0.2, 0.5, 20.0, 45.0), response_water_level=None, ) ``` When response processing is enabled, each returned waveform item includes an `instrument_processing` metadata dict with the matched native `response_id`, response epoch, optional simulated target response id, processing flags, and any error message. ### 9.2 Wildcard Query Mechanism `*` is converted to SQL `LIKE` (`%`). Supported: `*` (match all), `BK` (exact), `B*` (prefix), `*Z` (suffix), `BH*` (channel family). ### 9.3 Build SQLite Index ```bash python utils/hdf5_waveform_index.py build \ --h5 "data/hdf5/continuous_waveform_usa_*.h5" \ --db data/index/waveform_index.sqlite ``` On external drives, prefer the wrapper below. It builds the SQLite database on local temporary storage first and then copies the finished index into `data/index/`: ```bash ./scripts/build_waveform_index.sh ``` ### 9.4 Example Script ```bash python scripts/example_query_waveform.py \ --db data/index/waveform_index.sqlite \ --network BK --station BDM \ --starttime 2019-07-01T00:00:00 \ --endtime 2019-07-01T01:00:00 ``` Add response removal or target-response simulation when amplitude-aware access is needed: ```bash python scripts/example_query_waveform.py \ --db data/index/waveform_index.sqlite \ --network BK --station BDM --channel BHZ \ --starttime 2019-07-01T00:00:00 \ --endtime 2019-07-01T00:01:00 \ --remove_response \ --response_pre_filt 0.2 0.5 20 45 \ --response_water_level none ``` ### 9.5 ⚠️ macOS External Drive Issue SQLite requires file locking and journal/WAL files. External drives (exFAT, NTFS, network drives) often lack reliable locking support, causing `sqlite3.OperationalError: attempt to write a readonly database` even when HDF5 writes succeed normally. **Solution (recommended):** keep the SQLite index on a local disk while HDF5 data stays on the external drive: ```bash # HDF5 on the T5 project drive --h5 /Volumes/T5/publish/data/hdf5/*.h5 # SQLite on local disk --db ~/waveform_index.sqlite ``` Test SQLite writability explicitly: ```bash python -c "import sqlite3; sqlite3.connect('/tmp/seismicx_sqlite_test.db').execute('CREATE TABLE t(a)')" ``` --- ## 10. Phase Picking: `scripts/run_picker_to_jsonl.py` Runs a TorchScript seismic phase picker over every sample in the DataLoader and writes results to a JSONL file. Supports Mac/MPS, CUDA (single or multi-GPU), and CPU with automatic device selection, periodic model reloading, and built-in resume. --- ### 10.1 Quick Start **Mac / Apple Silicon (MPS):** ```bash python scripts/run_picker_to_jsonl.py \ --h5_input 'data/hdf5/continuous_waveform_usa_*.h5' \ --output_jsonl data/picks/output.jsonl \ --picker_model pickers/pnsn.v1.jit \ --polar_model pickers/polar.onnx \ --device mps ``` **CUDA with multi-process prefetch:** ```bash python scripts/run_picker_to_jsonl.py \ --h5_input 'data/hdf5/continuous_waveform_usa_*.h5' \ --output_jsonl data/picks/output.jsonl \ --picker_model pickers/pnsn.v1.jit \ --polar_model pickers/polar.onnx \ --device cuda \ --batch_size 4 \ --num_workers 4 \ --prefetch_factor 2 ``` **Resume an interrupted run (default behavior):** Simply re-run the same command. The script scans the existing JSONL at startup, rebuilds the set of already-processed sample keys, and removes them from the DataLoader index before any waveform data is read. --- ### 10.2 CLI Arguments | Argument | Default | Description | | --- | --- | --- | | `--h5_input` | `data/hdf5/continuous_waveform_usa_*.h5` | HDF5 file, directory, or glob pattern | | `--output_jsonl` | `data/picks/output.jsonl` | Output JSONL file (appended in resume mode) | | `--picker_model` | `pickers/pnsn.v1.jit` | TorchScript phase picker (`.jit`) | | `--polar_model` | `pickers/polar.onnx` | Optional ONNX polarity model; pass `""` to disable | | `--device` | `auto` | Inference device: `auto`, `cpu`, `cuda`, or `mps` | | `--batch_size` | `1` | DataLoader batch size; samples are still processed one by one | | `--num_workers` | `0` | DataLoader worker processes; 0 = single-process (required for MPS) | | `--prefetch_factor` | `2` | Batches pre-loaded per worker (ignored when `num_workers=0`) | | `--multiprocessing_context` | `auto` | `auto` selects `spawn` on Linux; override with `spawn`, `fork`, or `forkserver` | | `--allowed_families` | `HH,BH,EH,HN` | Comma-separated channel families | | `--allowed_z_only_channels` | `EHZ` | Comma-separated Z-only channel codes | | `--allow_z_only` / `--no_z_only` | `True` | Include / exclude Z-only stations | | `--replicate_z_only` / `--no_replicate_z_only` | `False` | Opt in to replicating Z as [Z,Z,Z] | | `--target_sampling_rate` | `100.0` | Resample to Hz; use `-1` to disable | | `--min_confidence` | `0.0` | Minimum pick probability to write | | `--snr_window_sec` | `2.0` | SNR window length before/after pick (seconds) | | `--progress_interval` | `100` | Print progress every N samples | | `--flush_interval` | `1000` | Flush JSONL every N samples; `0` = flush only at end | | `--gc_interval` | `200` | Run `gc.collect` + empty device cache every N samples | | `--resume` / `--no_resume` | `True` | Enable / disable resume mode | | `--mps_empty_cache_interval` | `1` | Call `torch.mps.empty_cache()` every N samples | | `--cuda_empty_cache_interval` | `200` | Call `torch.cuda.empty_cache()` every N samples | | `--reload_model_interval` | `-1` | Reload TorchScript model every N samples; `-1` = auto (50 on MPS, 0 on CUDA/CPU) | | `--include_segments_metadata` | `False` | Include raw HDF5 segment dicts in loader output | | `--no_keep_h5_open` | — | Disable per-file HDF5 handle caching | | `--no_overlap_mask` | — | Disable overlap mask in segment merging | | `--h5_rdcc_nbytes` | `8388608` (8 MB) | HDF5 chunk-cache size per file handle | | `--max_samples` | `0` | Exit after N new samples (code 75); re-run with `--resume` to continue. `0` = no limit. See MPS restart loop below. | --- ### 10.3 Device-Specific Notes **MPS (Mac / Apple Silicon)** Use `--num_workers 0` (single-process). The MPS TorchScript allocator accumulates internal Metal buffer state with each `sess(xt)` call; `torch.mps.empty_cache()` does not fully reclaim it. The model is reloaded every 50 samples by default (`--reload_model_interval -1`) to reset allocator state. Increase the interval for very short waveforms if the reload overhead is noticeable; set `--reload_model_interval 0` to disable. For long runs (thousands of samples), even model reloads cannot fully reclaim the Metal allocator's process-level state. The recommended pattern is to let the OS reclaim memory on each process exit using `--max_samples` and a restart loop: ```bash while true; do python scripts/run_picker_to_jsonl.py \ --h5_input 'data/hdf5/continuous_waveform_usa_*.h5' \ --output_jsonl data/picks/output.jsonl \ --picker_model pickers/pnsn.v1.jit \ --polar_model pickers/polar.onnx \ --device mps --max_samples 200 [ $? -ne 75 ] && break done ``` The script exits with code **75** when `--max_samples` is reached and code **0** when all samples are finished. `--resume` is enabled by default, so each restart skips already-written samples at the dataset level (no waveform data is re-read from HDF5). **CUDA** Use `--num_workers 4` or more to overlap HDF5 I/O with GPU inference. The `spawn` context is selected automatically on Linux. CUDA allocator state does not accumulate across TorchScript calls, so model reload is disabled by default. **CPU** Single-process; no special memory management needed. --- ### 10.4 Resume Mechanism At startup the script passes `skip_jsonl=output_jsonl` to `HDF5WaveformDataset`. The JSONL is scanned in parallel using `ThreadPoolExecutor`. Each finished record's sample key is reconstructed from its stored fields (`h5_file`, `year_id`, `day_id`, `station_id`, `channel_family`, `channels`). Matching index entries are dropped from the DataLoader — their waveform data is never read from HDF5. On the second and subsequent runs, startup takes O(N_finished × JSON parse cost). For very large JSONL files (>10 GB) this may take 30–120 seconds but is far cheaper than re-running inference. --- ### 10.5 Output JSONL Format Each output line is a self-contained JSON record. Two `record_type` values are written: **`"phase_pick"`** — one record per detected phase: ```json { "record_type": "phase_pick", "phase_id": 0, "phase_name": "Pg", "phase_prob": 0.987, "phase_sample": 15234.0, "phase_relative_time": 152.34, "phase_time": "2019-07-01T00:02:32.340000Z", "polarity": "U", "polarity_prob": 0.912, "polarity_available": true, "snr": 12.4, "amplitude": 3.14e-05, "pre_std": 2.10e-06, "post_std": 2.60e-05, "pre_abs_p95": 4.11e-06, "post_abs_p95": 5.87e-05, "channels": ["BHE", "BHN", "BHZ"], "channel_family": "BH", "component_order": "E/N/Z or 1/2/3", "is_z_only": false, "z_only_replicated": false, "waveform_shape": [8640001, 3], "sampling_rate": 100.0, "original_sampling_rate": 40.0, "target_sampling_rate": 100.0, "resampled": true, "h5_file": "data/hdf5/continuous_waveform_usa_20190701.h5", "year_id": "2019-01-01T00:00:00.000000Z", "day_id": "2019-07-01T00:00:00.000000Z", "station_info": { "station_id": "BK.BDM.00", "network": "BK", "station": "BDM", "location": "00", "longitude": -122.2486, "latitude": 37.8762, "elevation": 276.0, "location_available": true, "position_in_time_range": true, "position_is_fallback": false } } ``` **`"error"`** — one record per sample where inference failed: ```json { "record_type": "error", "station_id": "BK.BDM.00", "h5_file": "...", "year_id": "...", "day_id": "...", "sample_key": "...|...|...|...|BH|BHE,BHN,BHZ", "error": "RuntimeError: ..." } ``` **Phase ID → name mapping:** | `phase_id` | `phase_name` | | --- | --- | | 0 | Pg | | 1 | Sg | | 2 | Pn | | 3 | Sn | | 4 | P | | 5 | S | --- ## 11. Evaluating Picks: `essd_scripts/evaluate_picks.py` Matches automatically generated phase picks against provenance-preserving reference annotations, which include separate manual and operational automatic subsets. It uses a SQLite index to handle JSONL output files of 40 GB or more without loading them into memory. The optional matching records under `data/validation/multimodel_phase_matching/` are usability and interoperability checks for the data product and reporting scripts. They are not relative model- assessment evidence; their large source automatic-pick JSONL files are not part of the versioned extension. --- ### 11.1 Workflow **Step 1 — Build the SQLite index (first run only):** ```bash python essd_scripts/evaluate_picks.py \ --auto-jsonl data/picks/output.jsonl \ --index-db ~/output.index.sqlite \ --build-index \ --drop-existing ``` This streams through the JSONL, batch-inserts picks into SQLite (50,000 rows per commit), and builds indices on `(station_id, phase_name, sec_key)` and `time_epoch` for fast lookup. The index file is typically 3–5× smaller than the JSONL. Use `--drop-existing` when rebuilding an index after updating the source JSONL, so old picks are not retained. **Step 2 — Evaluate recall and residuals:** ```bash python essd_scripts/evaluate_picks.py \ --auto-jsonl data/picks/output.jsonl \ --label-json data/label/annotations_for_continuous_hdf5.json \ --reference-db data/label/reference_arrivals.sqlite \ --index-db ~/output.index.sqlite \ --outdir eval_picks/eval_phasenet \ --tp-tol 1.5 \ --err-window 5.0 ``` **Step 3 — Add an optional catalogue-conditioned reference-window output diagnostic:** ```bash python essd_scripts/evaluate_picks.py \ --auto-jsonl data/picks/output.jsonl \ --label-json data/label/annotations_for_continuous_hdf5.json \ --reference-db data/label/reference_arrivals.sqlite \ --index-db ~/output.index.sqlite \ --outdir eval_picks/eval_phasenet \ --tp-tol 1.5 \ --coverage-level C0 \ --output-eligibility-db data/label/output_eligibility.sqlite \ --output-name phasenet \ --reference-window-s 5 \ --reference-window-s 30 ``` Each `--reference-window-s` value is a half-width `T` and must be at least the one-to-one matching tolerance `epsilon = --tp-tol`. The utility applies the same declared C0--C3 configuration used for recall, retains references inside the selected output's processing domain, merges `+/-T` windows by exact station-location and phase, and intersects them with that processing domain. It reports the catalogue-conditioned match fraction together with raw and effective processed-window durations, eligible reference count, matched count, and catalogue-unmatched count. This is a precision-oriented output diagnostic, not an estimator of conventional full-stream precision. Both `T` and `epsilon`, the coverage level, and the output eligibility domain must be reported. **Step 4 — Generate figures (optional, requires matplotlib + scipy):** ```bash python essd_scripts/evaluate_picks.py \ --auto-jsonl data/picks/output.jsonl \ --label-json data/label/annotations_for_continuous_hdf5.json \ --reference-db data/label/reference_arrivals.sqlite \ --index-db ~/output.index.sqlite \ --outdir eval_picks/eval_phasenet \ --plot ``` **Inspect index statistics:** ```bash python essd_scripts/evaluate_picks.py \ --index-db ~/output.index.sqlite \ --db-info ``` --- ### 11.2 CLI Arguments | Argument | Default | Description | | --- | --- | --- | | `--auto-jsonl` | `data/picks/skynet.phase.jsonl` | JSONL output from `scripts/run_picker_to_jsonl.py` | | `--label-json` | `data/label/annotations_for_continuous_hdf5.json` | Annotation JSON (CEED format) | | `--reference-db` | `None` | Versioned C0--C3 flags from `reference_arrivals.sqlite`; mutually exclusive with `--waveform-db` | | `--index-db` | `~/skynet.pick.index.sqlite` | Path to SQLite index (keep on local disk — see Section 9.5) | | `--outdir` | `eval_picks/eval_skynet` | Directory for output files | | `--build-index` | flag | Stream JSONL and build / update SQLite index | | `--drop-existing` | flag | Drop the existing index table before rebuilding | | `--keep-raw-json` | flag | Store full raw JSON in SQLite (not recommended for large files) | | `--batch-size` | `50000` | SQLite insert batch size during indexing | | `--tp-tol` | `1.5` | True-positive tolerance window in seconds | | `--err-window` | `5.0` | Wider search window for residual distribution in seconds | | `--coverage-level` | `C0` | Coverage configuration used for recall and reference-window reporting (`C0`--`C3`) | | `--output-eligibility-db` | none | SQLite station-time processing domain required for reference-window reporting | | `--output-id`, `--output-name` | none | Exact assessed-output selector; one is required for reference-window reporting | | `--reference-window-s` | none | Half-width `T` for the optional catalogue-conditioned reference-window match fraction; repeat to report sensitivity | | `--min-prob` | `None` | Minimum `phase_prob` to include automatic picks | | `--phase-map` | `None` | Map label phases to automatic phases (see below) | | `--plot` | flag | Generate residual histograms and recall bar charts | | `--db-info` | flag | Print pick counts from index and exit | --- ### 11.3 Catalogue-conditioned reference-window match fraction For exact station-location `s` and normalized phase `phi`, the utility first selects manual references satisfying the declared C0--C3 coverage configuration and lying inside the assessed output's eligibility intervals. It merges `[t_r-T, t_r+T]` windows and intersects the union with those same eligibility intervals. The numerator is the number of deterministic one-to-one matches within `epsilon`; the denominator is every automatic pick in the resulting effective processed-window domain. Other denominator picks are catalogue- unmatched, not confirmed false detections. This fraction is strongly reference-conditioned when `T` is small. As `T` increases, the union covers more continuous output, but catalogue incompleteness becomes more consequential. It is therefore an output-audit diagnostic rather than conventional precision. Reuse requires the same coverage, waveform, reference, and workflow processing domains, preprocessing, threshold, phase mapping, `T`, `epsilon`, and one-to-one matching rule. Report both raw reference-window duration and effective processed-window duration. --- ### 11.4 Phase Map The `--phase-map` argument controls how label phase names are matched to automatic phase names. The default normalization is: ``` P → P, Pg, Pn S → S, Sg, Sn Pg → Pg Sg → Sg Pn → Pn Sn → Sn ``` Assignments are one-to-one within station, normalized phase, and label subset; one automatic-pick row cannot satisfy two reference arrivals. The time-ordered assignment maximizes matches within the declared TP tolerance before considering wider diagnostic-window assignments and total absolute residual. Override with a semicolon-separated list: ```bash # Match label P to Pg or Pn; label S to Sg or Sn --phase-map "P:Pg,Pn;S:Sg,Sn" # Strict: label P must match Pg only --phase-map "P:Pg;S:Sg" ``` --- ### 11.5 Annotation JSON Format The label file follows a nested structure: ``` years → days → events → stations → picks[] ``` Each pick entry must have at minimum: ```json { "phase": "P", "time": "2019-07-01T00:02:32.340000Z", "status": "manual", "station_id": "BK.BDM.00" } ``` Optional fields: `network`, `station`, `location`, `event_id`, `distance_km`. --- ### 11.6 Output Files All outputs are written to `--outdir`: | File | Description | | --- | --- | | `summary.json` | Per-phase and per-subset recall and residual statistics, automatic-pick counts, and optional catalogue-conditioned reference-window match fractions with declared coverage and workflow domains | | `summary.tsv` | Same data as tab-separated text, one row per (subset, phase) | | `matches.jsonl` | One JSON record per label pick: whether it matched, residual in seconds, auto phase probability, SNR | | `figures/residual_fit_{subset}_{phase}.png` | Residual histogram overlaid with fitted Gaussian and Student-t PDFs | | `figures/recall_bar.png` | Recall by phase and subset | | `figures/auto_pick_count_bar.png` | Automatic pick counts grouped by phase name | **Subsets reported:** `all` (every label pick), `manual` (manually reviewed only), `automatic` (automatically labeled only). **`summary.json` structure (excerpt):** ```json { "tp_tolerance_s": 1.5, "residual_window_s": 5.0, "label_phase_count_all_status": {"P": 1240, "S": 1198}, "subsets": { "all": { "P": { "n_label": 1240, "n_matched_within_tp_tol": 1102, "recall": 0.889, "residual_mean_s": 0.031, "residual_std_s": 0.214, "residual_median_s": 0.018, "residual_abs_p90_s": 0.45, "gaussian_fit_all_within_err_window": {"mean": 0.031, "std_mle": 0.213, "std_unbiased": 0.213}, "student_t_fit_all_within_err_window": {"df": 4.2, "loc": 0.029, "scale": 0.18} } } }, "auto_pick_count": { "total": 458320, "by_auto_phase": {"Pg": 231044, "Sg": 198210, "Pn": 18700, "Sn": 10366}, "mapped_to_label_phase": {"P": 231044, "S": 198210} }, "reference_window_match": [{ "window_half_width_s": 5.0, "matching_tolerance_epsilon_s": 1.5, "reference_subset": "C0-covered manual arrivals", "coverage_level": "C0", "workflow_eligibility": { "source": "data/label/output_eligibility.sqlite", "output_id": 1, "output_name": "phasenet", "valid_input_required": true }, "by_phase": { "P": { "n_coverage_qualified_manual_references": 1020, "n_workflow_eligible_manual_references": 1005, "n_merged_reference_windows": 810, "n_effective_processed_windows": 815, "reference_window_union_s": 9650.0, "effective_processed_window_s": 9400.0, "n_reported_picks_in_effective_windows": 1180, "n_matched_one_to_one": 910, "n_catalogue_unmatched_in_effective_windows": 270, "reference_window_match_fraction": 0.771, "reference_recovery_fraction": 0.905 } } }] } ``` The numbers in this JSON fragment are illustrative schema values, not released workflow results. --- ## 12. REAL Association: `scripts/run_real_association.py` `scripts/run_real_association.py` connects the SeismicX-Cont picker JSONL format to the REAL association algorithm. It compiles `scripts/real/real.c`, normalizes picker phases to P/S, writes REAL input files, runs association-based event detection, and exports a workflow JSONL containing both time-window phase candidates and associated earthquake events. The output can be used directly for event-detection evaluation or passed to a user-supplied earthquake-location program through the associated-arrival list. ```bash python scripts/run_real_association.py \ --picks-jsonl data/picks/pnsn_v3_diff.phase.jsonl \ --output-jsonl data/associated/real_events.jsonl \ --starttime 2019-07-01T00:00:00Z \ --endtime 2019-07-08T00:00:00Z ``` The output `data/associated/real_events.jsonl` includes one `real_association_run` metadata record, one `real_input_pick` record for each phase pick that entered REAL after filtering/NMS, and one `real_event` record for each associated event. Event arrivals keep a `source_pick.real_input_pick_id` so users can trace each associated phase back to the time-window phase record and then to the original picker JSONL line, station metadata, probability, channel family, and HDF5 file. The companion `real_events.summary.json` records filter counts and REAL run commands. The default settings are conservative workflow defaults rather than calibrated catalog-production parameters. Tune `--min-prob`, `--real-s`, `--real-r`, and `--real-g` for a particular picker, period, and velocity model. Detailed usage notes are in `scripts/README_real_association.md`. The association JSONL is intentionally simple. A downstream locator can ignore the REAL-specific diagnostic fields and consume only `real_event.picks[]`, where each associated arrival gives station identity, phase name, pick time, and a link back to the original automatic-pick record. Preliminary `latitude`, `longitude`, and `depth_km` fields are included when the associator provides them, but they are not required for downstream location workflows. ### 12.1 Event-Level Catalog Comparison Use `essd_scripts/compare_associated_events.py` to compare associated events against the SeismicX-Cont catalog or any event JSONL following the documented schema: ```bash python essd_scripts/compare_associated_events.py \ --pred-jsonl data/associated/real_events.jsonl \ --catalog data/label/annotations_for_continuous_hdf5.json \ --outdir eval_events/real_vs_catalog ``` An event is counted as catalogue matched when it matches a reference event with epicentral distance error `< 20 km` and origin-time error `< 3 s`. The script reports catalogue-matched, catalogue-unmatched, and unrecovered catalogue-event counts together with catalogue match and recovery fractions and time/location/depth/magnitude error statistics for matched events. These catalogue-relative labels do not assign physical false-event status to unmatched workflow events. The generic event JSONL format is documented in `essd_scripts/README_event_comparison.md`. --- ## 13. License, Citation, and Checksums Licensing is layer-specific. See `LICENSE` and `LICENSE_CLARIFICATION.md` for the full terms. - Waveform sample arrays stored in the HDF5 files are processed and structured derivatives of source waveform records. Publication and redistribution of these arrays have been clarified with the relevant U.S. seismic data providers, and their use is limited to non-commercial scientific research under upstream-compatible data-use terms. - Except for the waveform sample arrays, all assets in the full and mini releases are provided under **CC BY 4.0**. This includes metadata schemas, coverage indexes, event and phase annotations, response metadata, manifests, validation summaries, automatic-pick and consensus-candidate products, figures, reuse documentation, code in `scripts/`, `essd_scripts/`, and `utils/`, and notebooks. Only the waveform sample arrays are excluded from the SeismicX-Cont CC BY 4.0 grant. They are not relicensed by SeismicX-Cont and remain subject to applicable source-provider terms. Source lineage and citation requirements also remain in force for source-attributed metadata and derived records. Users must cite SeismicX-Cont together with the original SCEDC, NCEDC, CEED, and contributing network sources as appropriate. The release version is recorded in `VERSION`; releases should also be tagged with the same version string, for example `v1.0.0`. Machine-readable citation metadata are provided in `CITATION.cff`. The main-release checksum and file-size manifests can be regenerated with: ```bash ./scripts/write_manifest.sh ``` The script writes `manifest_sha256.txt` and `manifest_filesizes.tsv`. Both files exclude local build products, manuscript files, generated figures, mini-release files, temporary files, picker outputs, and intermediate hourly subsets. The HuggingFace and ModelScope repositories are kept aligned to these same manifest files, so either hosting service can be checked against the same paths, SHA-256 hashes, and file sizes. ESSD manuscript tables and figures can be regenerated from released data products and evaluation outputs with: ```bash ./essd_scripts/reproduce_manuscript_outputs.sh ``` This command writes audit reports under `essd_scripts/outputs/` and rebuilds the manuscript figures under `figures/`. The manuscript-number consistency step runs only when `overleaf-essd/template.tex` or `essd/template.tex` is present. A data release downloaded without the manuscript source still runs the inventory, coverage, waveform-quality, and figure checks and records the manuscript check as skipped. --- ## Contact SeismicX is developed by: - **Ziye Yu** - yuziye@cea-igp.ac.cn - **Yuqi Cai** - caiyuqiming@foxmail.com - **Xin Liu** - xinliu_geo@outlook.com - **Yanru An** - **Fajun Miu** - **Tengchao Dong** - **Tingting Li** - 972516012@qq.com - **Xinghui Huang** - huangxinhui@seis.ac.cn - **Lu Li** - **Ming Ma** - 925100762@qq.com ---Built with ❤️ for the seismology community