An IIoT application runs reliably for weeks. Sensors and controllers send their measured values to an edge gateway, which forwards the data to a historian, SCADA system or cloud platform. Then the WAN connection fails. The machine continues to operate and measured values continue to be generated – but the higher-level system is unavailable for several hours or even days.
Whether a complete data history is available after the connection is restored now depends largely on local buffering. An edge system requires sufficient persistent storage to retain all relevant data during the assumed offline period. At the same time, it must be clearly defined which data is stored, how large an individual data record actually is, which metadata is included and how the accumulated data will subsequently be retransmitted.
Buffer sizing therefore cannot be answered meaningfully with a general statement such as “a few gigabytes will be enough”. The required storage capacity results from the actual stored data rate, the maximum planned offline duration, storage and database overhead, and an appropriate reserve. It must also be checked whether the communication link is fast enough after an outage to reduce the accumulated backlog.
The key design rule is: An edge buffer must not only be large enough to survive an outage. The entire store-and-forward chain must be designed so that timestamps, data quality and sequence are preserved and the accumulated backlog can be retransmitted in a controlled manner once the connection has been restored.
Why edge buffering is required
A cloud or server connection can never be guaranteed absolutely in an industrial installation. Routers may restart, VPN connections may drop, mobile networks may fail, firewalls or certificates may cause problems, and planned maintenance work can also interrupt communication temporarily.
This does not necessarily create a problem for the actual acquisition of measured values. Sensors, field devices and controllers can continue operating locally. The outage only becomes critical if the architecture assumes that every measured value must be transmitted to the higher-level system immediately after it is generated.
An edge gateway with local buffer storage therefore decouples data acquisition from the WAN or cloud connection. The local side can continue acquiring and storing data while transmission is temporarily interrupted. Once the connection has been restored, data records that have not yet been transmitted are forwarded retrospectively.
A typical architecture can be simplified as follows:
Sensor / PLC → Edge gateway → local buffer → MQTT / HTTPS → Historian / Cloud
The local buffer should not be confused with a long-term historian. Its primary purpose is to bridge communication interruptions in a controlled manner and retain data until successful transmission has been confirmed by the target system.
From the measuring point to the actual storage data rate
The first step in sizing is to determine the actual data rate. A common mistake is to consider only the number of measured values and the size of the numerical value itself. A floating-point value, for example, requires only a few bytes. However, a data record that is useful in an industrial environment usually contains considerably more information.
In addition to the measured value itself, the stored data may include a measuring point ID, timestamp, unit, quality status, diagnostic information, sequence numbers and other metadata. Depending on the architecture, the structure of the data format and storage or database overhead must also be included.
| Variable | Meaning | Influence on sizing |
|---|---|---|
| N | Number of stored data points | More measuring points increase the data rate approximately proportionally |
| f | Storage frequency per data point | 1 Hz means, for example, one stored value per second |
| B | Effective number of bytes per stored data point | Includes value, identifier, timestamp, status and data format |
| t | Planned offline duration | Determines how long data must be retained locally without a connection |
| kO | Storage/database overhead | Includes, for example, indexes, journals, file structures and internal metadata |
| kR | Reserve factor | Provides margin for growth and deviations from the planning assumptions |
For an initial estimate, the storage data rate can be calculated using the following relationship:
Rstore = N × f × B
Here, Rstore is the local storage data rate, for example in bytes per second.
The decisive variable is B. Wherever possible, it should not be derived solely from the sensor’s raw payload size, but from the data format that is actually stored. A data record stored as a binary block can be considerably smaller than the same measured value stored in a detailed JSON structure containing a tag name, timestamp and status information.
Why the data model strongly affects storage requirements
The size of an individual process value tells us very little about the size of the stored data record. A temperature value may technically be represented as a 32-bit floating-point number and therefore require four bytes. As soon as a timestamp, measuring point identifier and quality status are added, however, the required storage space can increase several times over.
The difference becomes even more apparent with text-based formats. A structure such as:
{"tag":"pressure_01","timestamp":"2026-09-18T08:30:00.000Z","value":6.42,"quality":"Good"}
requires considerably more storage space than the numerical value alone. If every individual data point is stored in this way, the required buffer size increases accordingly.
This does not mean, however, that the most compact possible data format is always the best solution. For later usability, the measuring point identifier, original timestamp and quality status may be far more important than saving a few bytes. The objective should therefore not be a minimal data record, but a clearly defined data model that contains all information required by the application.
| Calculation assumption | Storage data rate | Data volume per 24 h |
|---|---|---|
| 20 data points, every 10 s, 64 bytes per point | 128 bytes/s | approx. 11.1 MB |
| 100 data points, 1 Hz, 64 bytes per point | 6.4 kB/s | approx. 553 MB |
| 500 data points, 2 Hz, 64 bytes per point | 64 kB/s | approx. 5.53 GB |
| 100 data points, 1 Hz, 150 bytes per point | 15 kB/s | approx. 1.30 GB |
The table shows how strongly the number of measuring points, storage frequency and data record size affect storage requirements. Particularly in large IIoT installations, seemingly small changes to the recording strategy can generate several additional gigabytes of data per day.
Basic formula for sizing
Once the effective storage data rate is known, the pure data volume generated during a communication interruption can first be calculated simply:
Mraw = Rstore × toffline
This value represents only the theoretical payload volume. For technical sizing, it should also be taken into account that databases, journaling, indexes, file systems and queue structures may require additional space.
A practical planning formula is therefore:
Mplan = Mraw × kO × kR
Here, kO represents the storage overhead that is actually expected and kR represents the desired planning reserve.
Wherever possible, these factors should be determined using a test system or real data recording. General percentage values are suitable only for an initial estimate. With databases in particular, data structure, indexing, compression, journaling and the number of small files can have a significant influence on the actual storage requirement.
Practical example: 72 hours of offline operation
A machine builder wants to record 120 process values using an edge system. Each data point is stored once per second. Testing has shown that a persisted data record, including timestamp, measuring point identifier and quality information, requires an average of approximately 80 bytes.
The resulting storage data rate is:
Rstore = 120 × 1 × 80 bytes/s
Rstore = 9,600 bytes/s
This results in approximately the following amount of data per day:
9,600 × 86,400 = 829,440,000 bytes
For approximate planning, this corresponds to around 829 MB per day.
The installation must be able to bridge a maximum outage of 72 hours, or three days, without losing data:
Mraw = 829 MB × 3 ≈ 2.49 GB
Testing shows that the database and queue require approximately 20% additional space. For planning purposes, an example reserve of 30% is also included:
Mplan = 2.49 GB × 1.20 × 1.30
Mplan ≈ 3.88 GB
For this application alone, approximately 3.9 GB of usable buffer storage would therefore be a calculated planning value. However, this does not mean that a 4 GB system partition would be sufficient. The operating system, applications, logs, updates and other local data require their own storage space. The storage allocated for store-and-forward should therefore be clearly distinguished technically from the remaining free storage available on the system.
Planning the storage reserve correctly
A storage reserve is important because the data rate in real installations is rarely completely constant. Additional diagnostic values may be added, software versions may change the data format and faults may generate more events and alarms than normal operation.
However, the reserve should not be selected arbitrarily. A better approach is to record the most realistic possible measurement data over a representative period and use this to determine the actual increase in storage usage. A reserve can then be added according to the system risk, planned operating period and expected future expansion.
The system should also not wait until the storage medium is completely full before reacting. In addition to the actual loss of data, a full system partition can cause further problems: databases may no longer be able to write journals, applications may crash and even system services may be affected.
| Offline duration | Raw data for 100 points, 1 Hz, 64 bytes | Planning value with 1.25 overhead factor and 30% reserve |
|---|---|---|
| 8 hours | approx. 184 MB | approx. 300 MB |
| 24 hours | approx. 553 MB | approx. 899 MB |
| 72 hours | approx. 1.66 GB | approx. 2.70 GB |
| 7 days | approx. 3.87 GB | approx. 6.29 GB |
These values are deliberately provided as calculation examples and should not be interpreted as universally applicable storage recommendations. The decisive factor is always the data model that is actually used.
What offline duration should be assumed?
The planned offline duration should not be derived solely from the technical expectation of the internet connection. What matters is the longest realistic period during which the target system may be unavailable.
In a production plant with its own on-call IT support, a communication problem may be resolved within a few hours. At a remote measuring station, a mobile network site or a plant that runs unattended over a weekend, several days may be a more realistic assumption.
Planned work should also be considered. Maintenance of firewall rules, VPN systems or cloud services can, for example, cause an interruption even though the local installation itself is operating correctly.
For planning purposes, the question should therefore not be: “How long is our connection normally unavailable?” but rather: “What offline duration must we be able to handle reliably without losing data?”
Distinguishing sampling rate, deadband and Change-of-Value
When calculating storage requirements, it is important to distinguish between measurement frequency, storage frequency and transmission frequency. A sensor may, for example, measure internally at a high frequency while the edge system stores only one value per second. Likewise, an edge system may record data locally at 1 Hz but send multiple measured values to the cloud in batches only every 30 seconds.
Change-of-Value and deadband strategies can also reduce the amount of data significantly. With slowly changing process variables, it is not always necessary to store or transmit every cyclically read measured value again if the value has not changed to a relevant degree.
However, caution is required when sizing the buffer: a low average Change-of-Value data rate does not guarantee that the data rate will also remain low during a process disturbance. In dynamic installations in particular, a large number of changes may occur during a process event.
The buffer should therefore not be sized exclusively for particularly quiet normal operation. For critical applications, a realistic upper data rate or a technically plausible worst-case scenario provides a much more meaningful basis.
Preserving timestamps and data quality
Store-and-forward only works correctly if a measured value transmitted later can still be assigned to its original acquisition time.
If a pressure value stored during a three-hour network outage were given a new timestamp during the later upload, the higher-level system would interpret the value as having been generated at the time of retransmission. The historical process representation would therefore be incorrect.
A useful buffer entry should therefore retain at least the information relevant to the application, for example:
- unique measuring point identifier,
- measured value,
- original timestamp or source timestamp,
- quality or status information,
- unit or scaling information if required,
- sequence number or unique data record ID where applicable.
The quality status is particularly important when data is retransmitted later. A numerical value alone does not indicate whether it was actually measured recently, whether communication with the sensor was already faulty, or whether only the last known value was being retained.
Designing store-and-forward correctly
A robust edge buffer should not operate according to the principle of “write a file now and send it at some point later”. It should always be possible to determine clearly which data has already been transmitted successfully and which data is still pending.
A typical process begins by storing a data record persistently on the local system. The edge system then attempts to transmit it to the target. Only once the selected communication or application layer considers the record to have been processed successfully should it be removed from the pending queue or marked as transmitted.
Depending on the system, different mechanisms may be used for this purpose: local databases, persistent message queues, journals or specially implemented store-and-forward buffers. The specific name matters less than the behaviour in fault situations.
The system should define clearly in particular what happens after a restart, how duplicate data records are prevented or detected, and in which order old and new data are transmitted.
During long outages, it may also be useful to handle data of different priorities differently. Process alarms and status changes may, for example, be more important than high-frequency trend values. However, such a strategy must already be incorporated into the data model and queue logic.
Can the backlog be transmitted quickly enough after an outage?
A sufficiently large storage capacity solves only the first half of the problem. Once the connection has been restored, the accumulated data must also be transmitted while new live data continues to be generated.
The decisive factor is therefore the available transmission capacity above the ongoing live data stream.
In simplified form:
Rfrei = RUpload - RLive
Only this remaining portion of the available bandwidth can be used to reduce the buffered backlog.
The approximate catch-up time is then:
tReplay = MBacklog / Rfrei
Example: After a longer outage, 2 GB of data is waiting to be retransmitted. Under actual operating conditions, the connection can provide 150 kB/s for this application. At the same time, the new incoming live data, including protocol overhead, already requires 25 kB/s.
The remaining capacity available to reduce the backlog is therefore:
150 kB/s - 25 kB/s = 125 kB/s
Under idealised conditions, retransmitting the 2 GB backlog would therefore take approximately 4.4 hours.
If the usable upload rate were lower than or equal to the ongoing live data rate, however, the buffer would not become smaller even after the connection had been restored. The backlog could remain permanently or even continue to grow.
For this calculation, the stored data volume and the actually transmitted data volume must be considered on the same basis. MQTT, HTTPS or other protocol frames, encryption, batch size and possible compression may mean that the network data rate does not correspond exactly to the local storage data rate.
Persistent storage instead of a volatile RAM buffer
For brief communication jitter lasting only a few seconds, a RAM buffer may be sufficient. For planned store-and-forward functions covering hours or days, however, relying exclusively on volatile memory is problematic. If power to the edge device fails at the same time or the operating system restarts, any data that has not been persisted will be lost.
For robust offline buffering, relevant data records must therefore be stored on a persistent medium. Depending on the platform, this may be eMMC, SSD, an industrial SD storage device or another non-volatile storage system.
In addition to nominal capacity, write endurance should also be considered if data is written frequently. An application that continuously writes many small data records can place a different load on flash storage than an application that collects larger blocks and writes them in batches.
The behaviour of the file system and database in the event of an abrupt loss of power is equally important. A UPS or buffered power supply can form part of the overall concept, but it does not replace a consistent storage and database strategy.
IT/OT operation and buffer monitoring
A buffer should not remain invisible during operation. If the storage simply continues to grow silently in the background, a failed cloud connection may not be noticed until the local storage medium is almost full.
Useful operating parameters include, for example:
- current queue size,
- used and free buffer storage,
- age of the oldest data record that has not yet been transmitted,
- number of pending data records,
- current write and upload rates,
- connection status to the target system,
- time of the last successfully confirmed transmission.
Warning thresholds can be defined on this basis. A warning at, for example, 70 or 80% buffer utilisation may be far more useful than reacting only when the storage medium is completely full. The specific threshold must, however, be appropriate for the architecture and the operator’s response time.
Systematic planning and testing procedure
For new IIoT projects, buffer sizing can be carried out using a clearly structured procedure.
- Identify all relevant data points: Which process values, states, alarms and diagnostic information must be retained during offline operation?
- Define the storage frequency: Specify not only the sensor sampling rate, but the rate at which data is actually persisted.
- Define the data model: Specify measured value, timestamp, tag ID, status and other metadata.
- Measure the actual data record size: Where possible, do not estimate it; generate a representative dataset instead.
- Determine the storage data rate: Calculate or measure the amount of data generated per minute, hour or day.
- Define the maximum offline duration: Include technical faults, maintenance work and unattended periods.
- Include overhead: Take databases, queues, journals and the file system into account.
- Define a reserve: Allow for growth, additional tags and load peaks.
- Check replay performance: Ensure that the backlog can be reduced faster after reconnection than new data is being generated.
- Test the limit case: Physically disconnect the network and simulate the intended offline duration or a realistic test scenario.
- Test a restart: Verify that data not yet uploaded remains available after a controlled restart.
- Implement monitoring: Monitor buffer fill level and the age of the oldest stored data.
Such a test is significantly more meaningful than a purely theoretical assessment. At the same time, it can be verified whether the data appears in the target system in the correct sequence and with the original time reference after the connection is restored.
Common planning mistakes
Calculating only the actual measured value in bytes
A floating-point value alone does not represent the actual storage requirement. Timestamps, IDs, quality information, data formats and database structures can account for a much larger share.
Confusing sampling rate with storage rate
A sensor may measure significantly faster than values are actually stored. The persisted data rate is the relevant parameter for buffer sizing.
Considering only average normal operation
With event-based storage, a system fault can generate a particularly large amount of data. A buffer sized only for a quiet production state may be too small precisely when a critical event occurs.
Failing to allow for database and file system overhead
The sum of the payload data does not necessarily correspond to the storage space actually occupied.
Using the entire free system storage as the buffer
The operating system, logs, updates and applications still require free storage space. A full file system can impair the entire edge system.
Calculating only the offline period
Once the connection has been restored, the backlog must also be transmitted. If the available bandwidth is insufficient, the buffer may remain full permanently.
Assigning timestamps only during the later upload
This causes the original measurement time to be lost. The history in the target system may then be displayed with incorrect timing.
Using RAM only as a long-term buffer
Buffered data can be lost after a power failure or restart. Persistent storage is required for longer offline periods.
Failing to define what happens when the storage is full
Every architecture requires a clear strategy for this limit case. Should the oldest data record be discarded? Is discarding data permissible at all? Should the system generate an alarm or switch to another defined operating strategy? This decision should not be left to the random behaviour of a completely full file system.
IIoT solutions at ICS Schneider
ICS Schneider Messtechnik supports IIoT architectures from the industrial measuring point through edge and gateway levels to integration with higher-level IT, SCADA, historian or cloud systems.
Typical data chains can integrate field devices and sensors into an edge layer via interfaces such as Modbus RTU, IO-Link, HART, OPC UA or Ethernet. From there, measured data can, for example, be forwarded to other systems using MQTT or HTTPS.
For the buffering described here, it is important to check the actual gateway and software functionality available for the specific project. Not every device used as a gateway automatically provides the same functions for local databases, persistent queues, store-and-forward or storage management.
One example of an IIoT architecture offered by ICS is the Siemens SITRANS MS200 for vibration and temperature monitoring in combination with the SITRANS CC220 gateway. Another example is the Siemens IIoT weighing electronics 7MH4647-0KK00-0AA2 for applications with the SIMATIC IOT2050.
IIoT solutions at ICS Schneider
Further reading: Choosing Change-of-Value and deadband correctly
Conclusion
A reliable edge buffer should not be selected according to the nominal storage capacity of a gateway, but should be sized based on the actual data chain.
Data points × storage frequency × actual data record size × offline duration form the starting point for the calculation. Storage and database overhead as well as an appropriately defined reserve must then be added.
The period after an outage is equally important. The connection to the higher-level system must provide sufficient additional transmission capacity so that the stored backlog can be reduced while the ongoing live data stream continues.
For a technically robust store-and-forward solution, storage capacity, persistent storage, original timestamps, quality status, queue management, retransmission and operational monitoring must therefore be considered together. Looking only at the number of available gigabytes leaves a major part of the system design unaddressed.
The most reliable sizing method is based on real test data. Recording a representative amount of data locally, measuring the actual increase in storage usage and then testing a complete offline/online cycle provides a significantly more robust basis than a purely theoretical estimate.
FAQ on Edge Buffer Storage
How do you calculate the size of an edge buffer?
As a starting point, multiply the number of data points by the storage frequency and the effective data record size. This gives the storage data rate. The result is then multiplied by the desired offline duration and supplemented by database/file-system overhead and a planning reserve.
How large is a single IIoT measured value?
There is no universal answer. The pure numerical value may require only a few bytes, but the stored data record often also contains a measuring point identifier, timestamp, quality status and other metadata. JSON, databases and other storage formats create additional overhead.
Is the size of a floating-point value sufficient for storage calculations?
No. For realistic sizing, the size of the data record that is actually persisted should be used.
Should I use the average or maximum data rate?
A realistic average may be useful for non-critical estimates. If data loss must be reliably prevented, it should also be checked which data rate can occur during dynamic operating states or events.
What offline duration should an edge gateway be able to bridge?
This depends on the operating model, network, installation location and response time. The decisive factor is the longest communication interruption that the system must be able to handle without data loss according to the system concept.
Why do I need additional storage reserve?
Data rate and data model can change. Databases, journals and file systems also require additional storage space. A reserve prevents small deviations from immediately reaching the storage capacity limit.
Is a RAM buffer sufficient for store-and-forward?
RAM may be useful for very short communication interruptions. For reliable buffering over longer periods, however, data that has not yet been transmitted should be stored persistently so that it can survive a restart or power failure.
What is store-and-forward?
Store-and-forward means that data is first stored locally when the target system is unavailable. Once the connection has been restored, the stored data records are transmitted retrospectively.
Why must the original timestamp be stored?
This ensures that a data record transmitted later can still be assigned to the actual time at which the measurement was taken. The time of the later cloud transmission must not replace the original event time.
What happens if the connection is too slow after an outage?
If the available transmission rate is only slightly higher than the ongoing live data rate, the backlog will be reduced very slowly. If it is equal to or lower than the live data rate, the buffer may continue to fill even though the connection is working again.
Can Change-of-Value reduce the required storage capacity?
Yes. For slowly changing measured variables, deadband or Change-of-Value strategies can significantly reduce the number of stored data points. When sizing the buffer, however, it should be taken into account that considerably more changes may occur during dynamic process conditions.
Should the edge buffer be allowed to use the entire free storage medium?
As a rule, no. The operating system, applications, logs, databases and updates require their own free storage space. Clear limits and warning thresholds should be defined for the data buffer.
What should happen when the buffer is full?
The behaviour must be defined for the specific project. Depending on the importance of the data, the system may, for example, generate an alarm, discard the oldest information or trigger another defined operating strategy. An uncontrolled file system filling up completely should be avoided.
How do you test a store-and-forward solution in practice?
The connection to the target system should be interrupted in a controlled manner. It should then be verified whether data continues to be acquired locally, how the storage usage increases, whether a restart is tolerated and whether, after the connection is restored, all data records are correctly retransmitted with their original timestamps.