Flashlist persistence

[gladky]

There are 21 flashlists to be persisted. Test was performed with 2 s sleep time between data retrieval.

Disk space consumption

• binary json format (smile) - 0.9TB/month - (1 flashlist snapshot: 2.1MB)
• json format 3.9TB/month - (1 flashlit snapshot 9.1MB)

Flashlist sizes

Flashlist list (ordered by size):

• FMMInputDetail (19.0%)
• ferolTcpStream (11.0%)
• FMMInput (10.2%)
• ferolConfiguration (9.5%)
• diskInfo (7.7%)
• jobcontrol (7.3%)
• levelZeroFM_static (5.7%)
• ferolMonitoring (5.4%)
• RU (4.0%)
• ferolStatus (2.8%)
• BU (2.8%)
• tcds_pm_tts_channel (1.5%)
• FMMStatus (1.1%)
• levelZeroFM_subsys (<1%)
• FMMFEDDeadTime (<1%)
• EVM (<1%)
• levelZeroFM_dynamic (<1%)
• hostInfo (<1%)
• FMMPartitionDeadTime (<1%)

Measured Aug 25.

[gladky]

Flashlist persistence introduced in release 1.4.0

[gladky]

Currently for 21 flashlists: 38GB per day, 1.18T per month measured Oct 24 (based on data sample data sep-oct)

Flashlists sizes vary significantly (e.g. 24h of FEROL_CONFIGURATION vary from 1.6GB to 4.1GB, 24h of FEROL_TCP_STREAM vary from 4.5GB to 6.4GB).

1. 1.6T needed to persist 24 Oct - 5 Dec including stops (42 days)
2. 1.3T needed to persist 24 Oct - 5 Dec excluding stops (33 days)

[hsakulin]

FMMFEDDeadtime and FMMPartitionDeadTime are for storing to DB, only. We do not need them.
I would keep all other flash lists and add several tcds flash lists.

[gladky]

Ok, so this is the final list:

1. BU
2. EVM
3. FMM_INPUT
4. FMM_INPUT_DETAIL
5. FMM_STATUS
6. RU
7. FEROL_CONFIGURATION
8. FEROL_INPUT_STREAM
9. FEROL_MONITORING
10. FEROL_STATUS
11. FEROL_TCP_STREAM
12. FRL_MONITORING
13. HOST_INFO
14. LEVEL_ZERO_FM_DYNAMIC
15. LEVEL_ZERO_FM_STATIC
16. LEVEL_ZERO_FM_SUBSYS
17. JOB_CONTROL
18. DISK_INFO
19. TCDS_CPM_COUNTS
20. TCDS_CPM_DEADTIMES
21. TCDS_CPM_RATES
22. TCDS_PM_ACTION_COUNTS
23. TCDS_PM_TTS_CHANNEL

Please confirm, I will then release 1.5.0 with those flashlists.

[hsakulin]

Looks ok. ferolInputStream and ferolStatus were not on your original list but of course they should be recorded.

On 27 Oct 2016, at 11:20, Maciej Gladki [email protected] wrote:

Ok, so this is the final list:

1. BU
2. EVM
3. FMM_INPUT
4. FMM_INPUT_DETAIL
5. FMM_STATUS
6. RU
7. FEROL_CONFIGURATION
8. FEROL_INPUT_STREAM
9. FEROL_MONITORING
10. FEROL_STATUS
11. FEROL_TCP_STREAM
12. FRL_MONITORING
13. HOST_INFO
14. LEVEL_ZERO_FM_DYNAMIC
15. LEVEL_ZERO_FM_STATIC
16. LEVEL_ZERO_FM_SUBSYS
17. JOB_CONTROL
18. DISK_INFO
19. TCDS_CPM_COUNTS
20. TCDS_CPM_DEADTIMES
21. TCDS_CPM_RATES
22. TCDS_PM_ACTION_COUNTS
23. TCDS_PM_TTS_CHANNEL

Please confirm, I will then release 1.5.0 with those flashlists.

�
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or mute the thread.

[gladky]

Disk usage exceeded the estimate.

2016/11/21T17:20: Used: 5.0TB, Free: 745GB
2016/11/23T11:20: Used: 5.1TB, Free: 561GB

This is increase of 185GB in 43h. Equal to 4.3G/h, 103GB/d. Expected 38GB/d!

[gladky]

Here is full data plot. You can see where we did estimates E1, E2 and why we run out of space. The flashlists' sizes changes over time. For instance there were always peaks of 3.5GB/h but since Nov 4 it's most of the time around 3.5GB/h.