Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[auditbeat] fim: implement kprobes backend #37796

Merged
merged 43 commits into from
Feb 14, 2024
Merged
Changes from 1 commit
Commits
Show all changes
43 commits
Select commit Hold shift + click to select a range
cb01f59
feat: add helper funcs to get symbol info from /proc/kallsyms
pkoutsovasilis Jan 24, 2024
9417243
feat: introduce fixed executor that always runs funcs from the same o…
pkoutsovasilis Jan 24, 2024
bbca9e7
feat: add probe manager to handle building tracing kprobes from tk-bt…
pkoutsovasilis Jan 24, 2024
292b7b7
feat: define probe events with corresponding alloc and release funcs
pkoutsovasilis Jan 24, 2024
d585da1
feat: embed stripped btf files and add helper funcs to read them
pkoutsovasilis Jan 24, 2024
e4616c1
feat: add fsnotify, fsnotify_nameremove, fsnotify_parent and vfs_geat…
pkoutsovasilis Jan 24, 2024
625da6b
feat: implement path traverser to produce monitor events by walking a…
pkoutsovasilis Jan 24, 2024
6f35ab1
feat: implement directory entries cache
pkoutsovasilis Jan 24, 2024
31ec585
feat: implement event processor to process probe events and based on …
pkoutsovasilis Jan 24, 2024
ea61593
feat: implement event verifier that validates that the expected seque…
pkoutsovasilis Jan 24, 2024
f58e369
feat: add perfChannel to reduce tracing.PerfChannel boilerplate code …
pkoutsovasilis Jan 24, 2024
0c785ca
feat: implement monitor that ties together path traverser, perf chann…
pkoutsovasilis Jan 24, 2024
d8bf292
feat: implement probe verification at runtime and the creation of a n…
pkoutsovasilis Jan 24, 2024
097aa25
feat: implement event reader for kprobe-based file integrity module
pkoutsovasilis Jan 24, 2024
c4a9d9b
doc: update NOTICE.txt to include tk-btf license
pkoutsovasilis Jan 24, 2024
6ca359f
feat: add tests for non-recursive kprobe fim (#3)
Tacklebox Jan 31, 2024
07b927b
fix: remove existing file from cache when a move operation is overwri…
pkoutsovasilis Jan 31, 2024
045bf40
feat: introduce force_backend in for file integrity auditbeat module
pkoutsovasilis Jan 31, 2024
bd6bcfb
ci: add necessary volume mounts for kprobes backend in auditbeat dock…
pkoutsovasilis Jan 31, 2024
651b2f7
feat: add the instantiation of file integrity module with kprobes bac…
pkoutsovasilis Jan 31, 2024
d39b22f
doc: update CHANGELOG.next.asciidoc
pkoutsovasilis Jan 31, 2024
1509a1a
fix: address compilation issues for non-linux oses
pkoutsovasilis Jan 31, 2024
0469332
fix: correct folder permission for path traverser unit-test
pkoutsovasilis Jan 31, 2024
6308e8b
fix: build kprobe package and unit-tests only for linux
pkoutsovasilis Jan 31, 2024
c52743b
ci: extend test_file_integrity.py to test kprobes backend of file int…
pkoutsovasilis Jan 31, 2024
6ccd479
ci: extend TestNew in monitor to include actual file changes
pkoutsovasilis Jan 31, 2024
82a07be
ci: mark with nolint prealloc slices that can't be pre-allocated
pkoutsovasilis Feb 1, 2024
cb2b330
Merge remote-tracking branch 'beats/main' into pkoutsovasilis/kprobe_fim
pkoutsovasilis Feb 1, 2024
5c00c37
Merge remote-tracking branch 'beats/main' into pkoutsovasilis/kprobe_fim
pkoutsovasilis Feb 1, 2024
3ad318b
Merge remote-tracking branch 'beats/main' into pkoutsovasilis/kprobe_fim
pkoutsovasilis Feb 3, 2024
4650e5f
chore: inline defer funcs
pkoutsovasilis Feb 9, 2024
62ea807
fix: return the scanner error if any
pkoutsovasilis Feb 9, 2024
f1cff58
fix: remove redundant runtime os checks for linux
pkoutsovasilis Feb 9, 2024
5350596
doc: comment that dEntryCache is not thread-safe
pkoutsovasilis Feb 9, 2024
fe6453a
fix: set the appropriate verbosity of errors of watcher
pkoutsovasilis Feb 9, 2024
da84277
fix: check for scanner.Err and return err from parsing mountinfo lines
pkoutsovasilis Feb 9, 2024
c4d2edb
fix: remove redundant fim_backends list from test_file_integrity.py
pkoutsovasilis Feb 13, 2024
6cd08cb
Merge remote-tracking branch 'beats/main' into pkoutsovasilis/kprobe_fim
pkoutsovasilis Feb 13, 2024
e745e23
fix: gofumpt kprobes package
pkoutsovasilis Feb 13, 2024
4a12aa9
fix: highlight unused context in event processor
pkoutsovasilis Feb 13, 2024
d80fbf5
fix: increase interval period of wait_output as kprobes require more …
pkoutsovasilis Feb 13, 2024
bd8d23a
fix: proper formatting for auditbeat.reference.yml
pkoutsovasilis Feb 13, 2024
f1e51f4
fix: proper formatting for x-pack/auditbeat/auditbeat.reference.yml
pkoutsovasilis Feb 13, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
feat: implement probe verification at runtime and the creation of a n…
…ew monitor based on these
pkoutsovasilis committed Jan 25, 2024
commit d8bf29233403725e8a7b9e7da72327cce48a5179
21 changes: 20 additions & 1 deletion auditbeat/module/file_integrity/kprobes/monitor.go
Original file line number Diff line number Diff line change
@@ -21,8 +21,11 @@ import (
"context"
"errors"
"fmt"
"github.com/elastic/elastic-agent-libs/logp"
"sync/atomic"
"time"

"github.com/elastic/elastic-agent-libs/logp"
"github.com/elastic/go-perf"
)

type MonitorEvent struct {
@@ -71,6 +74,22 @@ type Monitor struct {
closeErr error
}

func New(isRecursive bool) (*Monitor, error) {
ctx := context.TODO()

validatedProbes, exec, err := getVerifiedProbes(ctx, 5*time.Second)
if err != nil {
return nil, err
}

pChannel, err := newPerfChannel(validatedProbes, 10, 4096, perf.AllThreads)
if err != nil {
return nil, err
}

return newMonitor(ctx, isRecursive, pChannel, exec)
}

func newMonitor(ctx context.Context, isRecursive bool, pChannel perfChannel, exec executor) (*Monitor, error) {

mCtx, cancelFunc := context.WithCancel(ctx)
142 changes: 142 additions & 0 deletions auditbeat/module/file_integrity/kprobes/monitor_test.go
Original file line number Diff line number Diff line change
@@ -20,12 +20,19 @@ package kprobes
import (
"context"
"errors"
"io"
"net/http"
"os"
"os/exec"
"path/filepath"
"runtime"
"testing"
"time"

"github.com/elastic/beats/v7/auditbeat/tracing"

"github.com/stretchr/testify/mock"
"github.com/stretchr/testify/require"
"github.com/stretchr/testify/suite"
"golang.org/x/sys/unix"
)
@@ -459,3 +466,138 @@ func (p *monitorTestSuite) TestRunEmitError() {

p.Require().NoError(m.Close())
}

func (p *monitorTestSuite) TestNew() {

if runtime.GOARCH != "amd64" && runtime.GOARCH != "arm64" {
p.T().Skip("skipping on non-amd64/arm64")
return
}

if runtime.GOOS != "linux" {
p.T().Skip("skipping on non-linux")
return
}

if os.Getuid() != 0 {
p.T().Skip("skipping as non-root")
return
}

m, err := New(true)
p.Require().NoError(err)

p.Require().NoError(m.Start())
p.Require().NoError(m.Close())
}

const kernelURL string = "https://cdn.kernel.org/pub/linux/kernel/v6.x/linux-6.6.7.tar.xz"

func downloadKernel(filepath string) error {
// Create the file
out, err := os.Create(filepath)
if err != nil {
return err
}
defer out.Close()

// Get the data
req, err := http.NewRequestWithContext(context.TODO(), http.MethodGet, kernelURL, nil)
if err != nil {
return err
}

resp, err := http.DefaultClient.Do(req)
if err != nil {
return err
}
defer resp.Body.Close()

// Write the body to file
_, err = io.Copy(out, resp.Body)
return err
}

func BenchmarkMonitor(b *testing.B) {
if runtime.GOARCH != "amd64" && runtime.GOARCH != "arm64" {
b.Skip("skipping on non-amd64/arm64")
return
}

if runtime.GOOS != "linux" {
b.Skip("skipping on non-linux")
return
}

if os.Getuid() != 0 {
b.Skip("skipping as non-root")
return
}

tmpDir, err := os.MkdirTemp("", "kprobe_bench_test")
require.NoError(b, err)
defer os.RemoveAll(tmpDir)

tarFilePath := filepath.Join(tmpDir, "linux-6.6.7.tar.xz")

m, err := New(true)
require.NoError(b, err)

errChan := make(chan error)
cancelChan := make(chan struct{})

seenEvents := uint64(0)
go func() {
defer close(errChan)
for {
select {
case mErr := <-m.ErrorChannel():
select {
case errChan <- mErr:
case <-cancelChan:
return
}
case <-m.EventChannel():
seenEvents += 1
continue
case <-cancelChan:
return
}
}
}()

require.NoError(b, m.Start())
require.NoError(b, m.Add(tmpDir))

err = downloadKernel(tarFilePath)

// decompress
require.NoError(b, err)
cmd := exec.Command("tar", "-xvf", "./linux-6.6.7.tar.xz")
cmd.Dir = tmpDir
err = cmd.Run()
require.NoError(b, err)

// re-decompress; causes deletions of previous files
cmd = exec.Command("tar", "-xvf", "./linux-6.6.7.tar.xz")
cmd.Dir = tmpDir
err = cmd.Run()
require.NoError(b, err)

time.Sleep(2 * time.Second)
close(cancelChan)
err = <-errChan
if err != nil {
require.Fail(b, err.Error())
}

require.NoError(b, m.Close())

// decompressing linux-6.6.7.tar.xz created 87082 files (includes created folder); measured with decompressing and
// running "find . | wc -l"
// so the dcache entry should contain 1 (tmpDir) + 1 (linux-6.6.7.tar.xz archive)
// + 87082 (folder + archive contents) dentries
require.Len(b, m.eProc.d.index, 87082+2)

b.Logf("processed %d events", seenEvents)
}
147 changes: 147 additions & 0 deletions auditbeat/module/file_integrity/kprobes/verifier.go
Original file line number Diff line number Diff line change
@@ -21,17 +21,68 @@ package kprobes

import (
"bytes"
"context"
"embed"
"errors"
"io/fs"
"os"
"strings"
"time"

"github.com/elastic/beats/v7/auditbeat/tracing"

tkbtf "github.com/elastic/tk-btf"
)

//go:embed embed
var embedBTFFolder embed.FS

func getVerifiedProbes(ctx context.Context, timeout time.Duration) (map[tracing.Probe]tracing.AllocateFn, executor, error) {

fExec := newFixedThreadExecutor(ctx)

probeMgr, err := newProbeManager(fExec)
if err != nil {
return nil, nil, err
}

specs, err := loadAllSpecs()
if err != nil {
return nil, nil, err
}

var allErr error
for len(specs) > 0 {

s := specs[0]
if !probeMgr.shouldBuild(s) {
specs = specs[1:]
continue
}

probes, err := probeMgr.build(s)
if err != nil {
allErr = errors.Join(allErr, err)
specs = specs[1:]
continue
}

if err := verify(ctx, fExec, probes, timeout); err != nil {
if probeMgr.onErr(err) {
continue
}
allErr = errors.Join(allErr, err)
specs = specs[1:]
continue
}

return probes, fExec, nil
}

fExec.Close()
return nil, nil, errors.Join(allErr, errors.New("could not validate probes"))
}

func loadAllSpecs() ([]*tkbtf.Spec, error) {
var specs []*tkbtf.Spec

@@ -83,3 +134,99 @@ func loadEmbeddedSpecs() ([]*tkbtf.Spec, error) {

return specs, nil
}

func verify(ctx context.Context, exec executor, probes map[tracing.Probe]tracing.AllocateFn, timeout time.Duration) error {
basePath, err := os.MkdirTemp("", "verifier")
if err != nil {
return err
}

defer func() {
_ = os.RemoveAll(basePath)
}()

verifier, err := newEventsVerifier(basePath)
if err != nil {
return err
}

pChannel, err := newPerfChannel(probes, 4, 512, exec.GetTID())
if err != nil {
return err
}

m, err := newMonitor(ctx, true, pChannel, exec)
if err != nil {
return err
}

defer func() {
_ = m.Close()
}()

// start the monitor
if err := m.Start(); err != nil {
return err
}

// spaw goroutine to send events to verifier to be verified
cancel := make(chan struct{})
defer close(cancel)

retC := make(chan error)

go func() {
defer close(retC)
for {
select {
case runErr := <-m.ErrorChannel():
retC <- runErr
return

case ev, ok := <-m.EventChannel():
if !ok {
retC <- errors.New("monitor closed unexpectedly")
return
}

if err := verifier.validateEvent(ev.Path, ev.PID, ev.Op); err != nil {
retC <- err
return
}
continue
case <-time.After(timeout):
return
case <-cancel:
return
}
}
}()

// add verify base path to monitor
if err := m.Add(basePath); err != nil {
return err
}

// invoke verifier event generation from our executor
if err := exec.Run(verifier.GenerateEvents); err != nil {
return err
}

// wait for either no new events arriving for timeout duration or
// ctx to be cancelled
select {
case err = <-retC:
if err != nil {
return err
}
case <-ctx.Done():
return ctx.Err()
}

// check that all events have been verified
if err := verifier.Verified(); err != nil {
return err
}

return nil
}