!test -f aircraft.bufr || wget https://sites.ecmwf.int/repository/pdbufr/examples/aircraft.bufr

How to use the prefilter_headers option?

A BUFR message consists of a header and a data section. The header is typically small and fast to load while accessing the data section requires ecCodes to unpack the message, which can take up significantly longer time.

Using the prefilter_headers=True option can result in a significant performance boost when read_bufr() is called with filters containing header keys. In this case the filters are evaluated on the header keys before the message is unpacked and if they do not pass the message is skipped and the data section is not loaded at all.

Extracting aircraft data

We demonstrate it by using a BUFR file containing around 30000 aircraft messages. We want to extract a particular flight from this data using the ident header key (please note it is only available in messages containing a local header section generated by ECMWF).

import pdbufr

First, we use the default options (prefilter_headers=False) and measure the execution time.

%%time
df = pdbufr.read_bufr("aircraft.bufr", columns=["aircraftFlightNumber"], filters={"ident": "EU4333"})
len(df)

CPU times: user 5.83 s, sys: 259 ms, total: 6.09 s
Wall time: 6.15 s

47

Next, we use prefilter_headers=True. We can see that the data extraction became about 20% faster.

%%time
df = pdbufr.read_bufr(
    "aircraft.bufr", columns=["aircraftFlightNumber"], filters={"ident": "EU4333"}, prefilter_headers=True
)
len(df)

CPU times: user 4.51 s, sys: 216 ms, total: 4.73 s
Wall time: 4.84 s

47