JD
Manual Data Ingestion System
System Administration
Presentation Mode: upload actions are non-destructive and shown for workflow demonstration only.
Manual Upload Intake
Supports GRIB2, netCDF, BUFR, HDF5, UF, UHRIT/HRIT, HSD, Gempak, ASCII, shapefile, GeoJSON, GeoTIFF, KML, KMZ.
Processing Workflow
1. Upload and Registration
User uploads source files and captures source metadata, timestamp, station/area, and data category.
2. Decode and Validation
System auto-detects format, validates schema/headers, and checks checksum, georeference, and temporal consistency.
3. Processing and Transformation
Data is transformed to KDSS canonical structures and prepared for APIs, analytics layers, and downstream dissemination.
4. Storage and Indexing
Validated outputs are written to HDF5 and SQL repositories with version tags and traceable ingestion logs.
5. Distribution and Interoperability
Processed datasets are exposed to visualization clients and interoperable endpoints including GeoRiskPH and NDRRMC dashboards.
System Components
- Data visualization client
- Server-side data ingestion, processing, and distribution engine
- Local Data Manager
- HDF5 and SQL-based data storage platform
- Data processing APIs
- Data Ingestion, Processing and Management core services
Authoritative Multi-Source Feeds
- Weather radar (Doppler Level II and Level III)
- Geostationary satellite data (Himawari-8/9, GK2A)
- Surface observation data (SYNOP, AWS, METAR)
- Upper-air sounding data (TEMP/RAOB)
- Lightning detection network data
- Numerical weather prediction models
- Oceanographic and marine weather data (wave and hydrodynamic models, tide station, buoys, ship reports, marine AWS, HF radar)
- Hydrological data (river stages, rainfall)
- Specialized prediction models (flood and storm surge)
Manual Entry and Quality Controls
- Manual station data entry forms for manned stations
- Batch CSV/TXT upload for delayed station submissions
- Duplicate detection and merge review before commit
- Quality control flags for outliers, missing values, and unit mismatch
- Operator acknowledgment and supervisor approval trail
Recent Manual Upload Queue
| Dataset | Source | Format | Uploader | Status |
|---|---|---|---|---|
| AWS_SYNOPTIC_REGION02_20260603_1000.csv | Surface Observation (AWS) | ASCII/CSV | Hydro Desk Operator | Processed |
| HIMAWARI9_IR_20260603_0930.hsd | Satellite | HSD | Satellite Unit | Decoded |
| CAGAYAN_BASIN_STAGE_20260603_0900.txt | Hydrological | ASCII text | River Forecast Center | Validated |
| barangay_flood_extent_jun03.geojson | External GIS | GeoJSON | LGU Analyst | Queued Review |
Supported Data Formats
GRIB2
netCDF
BUFR
HDF5
UF
UHRIT/HRIT
HSD
Gempak
ASCII text
Shapefile
GeoJSON
GeoTIFF
KML
KMZ
Interoperability Targets
| Platform | Integration Mode | Status |
|---|---|---|
| GeoRiskPH | API-based data exchange | Supported |
| NDRRMC Monitoring Dashboard | Curated feed and bulletin bridge | Supported |
| Observing Facilities Directory | Central directory synchronization | Integrated |