Nanosec accurate packet processing #796

plangarbalint · 2023-05-02T14:56:45Z

The aim of this change is to make packet processing possible with nanosecond accuracy and to handle sub sleep times by only nanosleep()'ing.

change struct timeval to struct timespec where it was necessary
to have backward compatibility with gettimeofday, get_current_time wrapper function is used to retrieve current time (either using gettimeofday or clock_gettime based on the value of _POSIX_C_SOURCE)
pcap_open_offline is change to pcap_open_offline_with_tstamp_precision in replay_file function, as a result nanosec accuracy is not lost now directly when pcap file is read
extend nanosleep_sleep function with clock_nanosleep feature using TIMER_ABSTIME in order to avoid relative sleep when using nanosleep()
extend loop delay feature with nanosec delaying in case --timer=nano is set, earlier only microsec delay was possible

…n compile time

…me_timefunction Use clock_nanosleep with TIMER_ABSTIME, decide time function in use i…

…ep method

Read pcap files with nanosec precision, set nano_sleep as default sle…

fklassen

Thanks for the PR. Will target for next major release.

fklassen · 2023-05-02T15:32:24Z

src/common/timer.c

 }

+int get_current_time(struct timespec *ts){
+    #if defined _POSIX_C_SOURCE  && _POSIX_C_SOURCE >= 199309L
+        return clock_gettime(CLOCK_REALTIME, ts);


Haven't looked at this closely, but would this work with CLOCK_MONOTONIC? It would not be susceptible to NTP, but would not work if real-time clocks are needed.

using CLOCK_MONOTONIC is also possible, I fixed it in a new commit. I also had to modify gettimeofday_sleep function to work properly with CLOCK_MONOTONIC

Since the PR also using absolute time semantics for more precise sleeping, both CLOCK_MONOTONIC and CLOCK_REALTIME would work. However AFAIK POSIX only defines CLOCK_REALTIME mandatory. Probably every sane OS implements both clocksource, but I think realtime chosen here because of portability concerns.

I'll put it through it's paces when I start work on versin 4.5. I'll see if I can get my confidence up that it won't affect IoT and oldert stuff.

fklassen · 2023-05-02T15:34:07Z

src/common/timer.c

+        struct timeval tv;
+        int success = gettimeofday(&tv, NULL);
+        TIMEVAL_TO_TIMESPEC(&tv, ts);
+        return success;


nice that this supports older OS's. How much testing has there been on this?

A comprehensive test on this is definitely missing, I made some user tests with basic setups + all the tests pass when make test is executed. However there are many include guards, with if, elseif etc... branches, they all need to be tested separately. Just to mention one, it wasn't tested how it works with netmap

…_until

…eep to work with CLOCK_MONOTONIC

fklassen · 2023-09-03T23:33:58Z

src/send_packets.c

             *
             * ensure there is no overflow in cases where bits_sent is very high
             */
            if (bits_sent > COUNTER_OVERFLOW_RISK && bps > 500000)
-                next_tx_us = (bits_sent * 1000) / bps * 1000;
+                next_tx_ns = (bits_sent * 1000) / bps * 1000000;


This will take some amount of testing. Some overflows take > 1 week to occur.

…_accurate_packet_processing

fklassen · 2023-12-24T20:55:39Z

Merging to working branch

format commit with `git-clang-format HEAD~1`

* macOS was not reading nanosecond timer * get time called too often, causing overhead * fine tune gettimeofday_sleep() * adjusted overflow value to prevent issues with very long running instances * fix build/merge issues

…rocessing Feature #796 - nanosecond packet processing

Recently new version of tcpreplay was released[1]. This version now uses nanosecond timestamp precision[2][3] instead of microsecond precision. However, function datetime.datetime.fromisoformat() in Python up to 3.10 doesn't accept this format and fails with following error: tools/ft-orchestration/src/generator/tcpreplay.py:262: in stats start_time = datetime.datetime.fromisoformat(start_time) E ValueError: Invalid isoformat string: '1970-01-01 01:16:52.837054314' This commit fixes the issue by cropping nanosecond precision back to microsecond precision. Another solution would be to upgrade to Python 3.11, where fromisoformat() accepts nanosecond precision (even though it internally crops it back to microseconds). [1] https://github.com/appneta/tcpreplay/releases/tag/v4.5.1 [2] appneta/tcpreplay#796 [3] appneta/tcpreplay#863

eplabal and others added 5 commits March 6, 2023 14:18

rewrite do_sleep to handle sub sleep times by only nanosleeping

3b0934e

Use clock_nanosleep with TIMER_ABSTIME, decide time function in use i…

2997633

…n compile time

Merge pull request #1 from plangarbalint/timerabs_time_and_compile_ti…

a57b3e2

…me_timefunction Use clock_nanosleep with TIMER_ABSTIME, decide time function in use i…

Read pcap files with nanosec precision, set nano_sleep as default sle…

00e9f83

…ep method

Merge pull request #3 from plangarbalint/read_pcap_with_nanosec

ea08502

Read pcap files with nanosec precision, set nano_sleep as default sle…

fklassen reviewed May 2, 2023

View reviewed changes

plangarbalint added 4 commits May 2, 2023 21:12

fix gettimeofday_sleep function, add nap to now variable to get sleep…

a044d72

…_until

use CLOCK_MONOTONIC in clock_gettime function, modify gettimeofday_sl…

77a1448

…eep to work with CLOCK_MONOTONIC

fix time related bug in send_dual_packets function

6085e4f

change CLOCK_REALTIME to CLOCK_MONOTONIC flag in clock_nanosleep

ba8c807

fklassen reviewed Sep 3, 2023

View reviewed changes

fklassen changed the base branch from master to Feature_#796_nanosecond_packet_processing December 24, 2023 20:22

Merge branch 'Feature_#796_nanosecond_packet_processing' into nanosec…

b43ad7f

…_accurate_packet_processing

fklassen merged commit d5c8596 into appneta:Feature_#796_nanosecond_packet_processing Dec 24, 2023
1 check failed

fklassen added a commit that referenced this pull request Dec 25, 2023

Feature #796: nanosecond -changelog/credit

6af4341

fklassen added a commit that referenced this pull request Dec 25, 2023

Feature #796: clang-format mods

f9e2f70

format commit with `git-clang-format HEAD~1`

fklassen mentioned this pull request Dec 26, 2023

Feature #796 - nanosecond packet processing #836

Merged

fklassen added a commit that referenced this pull request Dec 26, 2023

Feature #796 - Nano timer test no longer requires --timer

3f1a891

fklassen added a commit that referenced this pull request Dec 29, 2023

Merge pull request #836 from appneta/Feature_#796_nanosecond_packet_p…

0dc684a

…rocessing Feature #796 - nanosecond packet processing

fklassen mentioned this pull request Jun 2, 2024

[Bug] regression with nanosecond timestamps #863

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Nanosec accurate packet processing #796

Nanosec accurate packet processing #796

plangarbalint commented May 2, 2023

fklassen left a comment

fklassen May 2, 2023

plangarbalint May 2, 2023 •

edited

Loading

SPYFF May 3, 2023 •

edited

Loading

fklassen Jun 5, 2023

fklassen May 2, 2023

plangarbalint May 2, 2023 •

edited

Loading

fklassen Sep 3, 2023

fklassen commented Dec 24, 2023

Nanosec accurate packet processing #796

Nanosec accurate packet processing #796

Conversation

plangarbalint commented May 2, 2023

fklassen left a comment

Choose a reason for hiding this comment

fklassen May 2, 2023

Choose a reason for hiding this comment

plangarbalint May 2, 2023 • edited Loading

Choose a reason for hiding this comment

SPYFF May 3, 2023 • edited Loading

Choose a reason for hiding this comment

fklassen Jun 5, 2023

Choose a reason for hiding this comment

fklassen May 2, 2023

Choose a reason for hiding this comment

plangarbalint May 2, 2023 • edited Loading

Choose a reason for hiding this comment

fklassen Sep 3, 2023

Choose a reason for hiding this comment

fklassen commented Dec 24, 2023

plangarbalint May 2, 2023 •

edited

Loading

SPYFF May 3, 2023 •

edited

Loading

plangarbalint May 2, 2023 •

edited

Loading