The new lzma2 compression options at the top of compress_blob_xz are
equivalent to using preset "0", exept for using a 1 MiB dictionary
(the same as preset "1"). This makes the memory usage at most 7.5 MiB
in the compressor, and 1 MiB in the decompressor, instead of the
previous 92 MiB in the compressor and 8 MiB in the decompressor.
According to test-compress-benchmark this commit makes XZ compression
20 times faster, with no increase in compressed data size.
Using more realistic test data (an ELF binary rather than repeating
ASCII letters 'a' through 'z' in order) it only provides a factor 10
speedup, and at a cost if a 10% increase in compressed data size.
But that is still a worthwhile trade-off.
According to test-compress-benchmark XZ compression is still 25 times
slower than LZ4, but the compressed data is one eighth the size.
Using more realistic test data XZ compression is only 18 times slower
than LZ4, and the compressed data is only one quarter the size.
$ ./test-compress-benchmark
XZ: compressed & decompressed 2535300963 bytes in 42.30s (57.15MiB/s), mean compresion 99.95%, skipped 3570 bytes
LZ4: compressed & decompressed 2535303543 bytes in 1.60s (1510.60MiB/s), mean compresion 99.60%, skipped 990 bytes
(cherry picked from commit 1930eed2a7)
Conflicts:
src/journal/compress.c
(cherry picked from commit 00464ad8a6)
(cherry picked from commit b4f5701719)
sd_j_e_u needs to keep a reference to an object while comparing it
with possibly duplicate objects in other files. Because the size of
mmap cache is limited, with enough files and object to compare to,
at some point the object being compared would be munmapped, resulting
in a segmentation fault.
Fix this issue by turning keep_always into a reference count that can
be increased and decreased. Other callers which set keep_always=true
are unmodified: their references are never released but are ignored
when the whole file is closed, which happens at some point. keep_always
is increased in sd_j_e_u and later on released.
(cherry picked from commit ae97089d49)
(cherry picked from commit 9e6366a959)
Conflicts:
src/journal/journal-verify.c
In the time it takes to process incoming log messages, the process we
are logging details for may exit. This means the cgroup data is no
longer available from '/proc'. Unfortunately, the way the code was
structured before, we never log _SYSTEMD_UNIT if we don't have this
cgroup information.
Add an else if case that allows the passed in unit_id to be logged even
if we couldn't capture cgroup information. This ensures a command like
`journalctl -u run-XXX` will return all log messages from a oneshot
process.
(cherry picked from commit 2d43b19090)
(cherry picked from commit 66d0783137)
Just as 'identifier' is strdup-ed and freed, we need to do the same for
unit_id.
(cherry picked from commit f92ae4968f)
(cherry picked from commit a345f4c7f5)
Since the journal can handle multiple lines just well natively,
and rsyslog can be configured to handle them as well, there is no need
to truncate messages from syslog() after the first newline.
Reproducer:
1. Add following four lines to /etc/rsyslog.conf
----------
$EscapeControlCharactersOnReceive off
$ActionFileDefaultTemplate RSYSLOG_SysklogdFileFormat
$SpaceLFOnReceive on
$DropTrailingLFOnReception off
----------
3. Restart rsyslog
# service rsyslog restart
4. Compile and run the following program
----------
#include <stdio.h>
#include <syslog.h>
int main()
{
syslog(LOG_INFO, "aaa%caaa", '\n');
return 0;
}
----------
Actual results:
Below message appears in /var/log/messages.
----------
Sep 7 19:19:39 localhost test2: aaa
----------
Expected results:
Below message, which worked prior to systemd-journald
appears in /var/log/messages.
----------
Sep 7 19:19:39 localhost test2: aaa aaa
https://bugzilla.redhat.com/show_bug.cgi?id=855313
"syslog(3) and sd_journal_print() may largely be used interchangeably
functionality-wise" according to sd_journal_print(3). This socket
should be always available except in rare circumstatances, and we
don't random applications to fail on logging, so let's do what syslog
did. The alternative of forcing all callers to do error handling for
this rare case doesn't really have any benefits, since if they can't
log there isn't much they can do anyway.
https://bugzilla.redhat.com/show_bug.cgi?id=1023041
Before, when the user journal file was rotated, journal_file_rotate
could close the old file and fail to open the new file. In that
case, we would leave the old (deallocated) file in the hashmap.
On subsequent accesses, we could retrieve this stale entry, leading
to a segfault.
When journal_file_rotate fails with the file pointer set to 0,
old file is certainly gone, and cannot be used anymore.
https://bugzilla.redhat.com/show_bug.cgi?id=890463
Vacuuming behaviour is a bit confusing, and/or we have some bugs,
so those additional messages should help to find out what's going
on. Also, rotation of journal files shouldn't be happening too
often, so the level of the messages is bumped to info, so that
they'll be logged under normal operation.
Before my previous patch, journal_file_empty wasn't be called with the
correct filename. Now that it's being called with the correct filename
it leaks file descriptors. This patch closes the file descriptors before
returning.
Signed-off-by: George McCollister <george.mccollister@gmail.com>
[Edit harald@redhat.com: make use of _cleanup_close_ instead]
d_name is modified on line 227 so if the entire journal name is needed
again p must be used. Before this change when journal_file_empty was called
on archived journals it would always return with -2.
Signed-off-by: George McCollister <george.mccollister@gmail.com>
Corrupted empty files are relatively common. I think they are created
when a coredump for a user who never logged anything before is
attempted to be written, but the write does not succeed because the
coredump is too big, but there are probably other ways to create
those, especially if the machine crashes at the right time.
Non-corrupted empty files can also happen, e.g. if a journal file is
opened, but nothing is ever successfully written to it and it is
rotated because of MaxFileSec=. Either way, each "empty" journal file
costs around 3 MB, and there's little point in keeping them around.
In case of scripts, _EXE is set to the interpreter name, and
_COMM is set based on the file name. Add a match for _COMM,
and _EXE if the interpreter is not a link (e.g. for yum,
the interpreter is /usr/bin/python, but it is a link to
/usr/bin/python2, which in turn is a link to /usr/bin/python2.7,
at least on Fedora, so we end up with _EXE=/usr/bin/python2.7).
I don't think that such link chasing makes sense, because
the final _EXE name is more likely to change.
Currently this check happens when the coredump has been collected in
it's entirety and being received by journald. this is not ideal
behaviour when the crashing process is consuming significant percentage
of physical memory such as a large instance of firefox or a java
application.
In 49998b383 (journald: do not overwrite syslog facility when
parsing priority) journald started ignoring facility part when
reading service stderr to convert to syslog messages. In this
case it is fine, because only the priority is allowed.
But the same codepath is used for syslog messages, where the
facility should be used. Split the two codepaths by explicitly
specyfing whether the facility should be ignored or not.
https://bugzilla.redhat.com/show_bug.cgi?id=988814
hashmap_free() wasn't being called on m->contexts and m->fds resulting
in a leak.
To reproduce do:
while(1) {
sd_journal_open(&j, SD_JOURNAL_LOCAL_ONLY);
sd_journal_close(j);
}
Memory usage will increase until OOM.
In the following scenario:
server creates system.journal
server creates user-1000.journal
both journals share the same seqnum_id.
Then
server writes to user-1000.journal first,
and server writes to system.journal a bit later,
and everything is fine.
The server then terminates (crash, reboot, rsyslog testing,
whatever), and user-1000.journal has entries which end with
a lower seqnum than system.journal. Now
server is restarted
server opens user-1000.journal and writes entries to it...
BAM! duplicate seqnums for the same seqnum_id.
Now, we usually don't see that happen, because system.journal
is closed last, and opened first. Since usually at least one
message is written during boot and lands in the system.journal,
the seqnum is initialized from it, and is set to a number higher
than than anything found in user journals. Nevertheless, if
system.journal is corrupted and is rotated, it can happen that
an entry is written to the user journal with a seqnum that is
a duplicate with an entry found in the corrupted system.journal~.
When browsing the journal, journalctl can fall into a loop
where it tries to follow the seqnums, and tries to go the
next location by seqnum, and is transported back in time to
to the older duplicate seqnum. There is not way to find
out the maximum seqnum used in a multiple files, without
actually looking at all of them. But we don't want to do
that because it would be slow, and actually it isn't really
possible, because a file might e.g. be temporarily unaccessible.
Fix the problem by using different seqnum series for user
journals. Using the same seqnum series for rotated journals
is still fine, because we know that nothing will write
to the rotated journal anymore.
Likely related:
https://bugs.freedesktop.org/show_bug.cgi?id=64566https://bugs.freedesktop.org/show_bug.cgi?id=59856https://bugs.freedesktop.org/show_bug.cgi?id=64296https://bugs.archlinux.org/task/35581https://bugzilla.novell.com/show_bug.cgi?id=817778
Possibly related:
https://bugs.freedesktop.org/show_bug.cgi?id=64293
Since 11ec7ce, journald isn't setting the ACLs properly anymore if
the files had no ACLs to begin with: acl_set_fd fails with EINVAL.
An ACL with ACL_USER or ACL_GROUP entries but no ACL_MASK entry is
invalid, so make sure a mask exists before trying to set the ACL.
Bump the minimal size of the journal so that we can be sure creating the
journal file will always succeed. Previously the minimum size was
smaller than a empty jounral file...
I'm assuming that it's fine if a _const_ or _pure_ function
calls assert. It is assumed that the assert won't trigger,
and even if it does, it can only trigger on the first call
with a given set of parameters, and we don't care if the
compiler moves the order of calls.
I typically run VMs with 1024MiB allocated; systemd is unable to write
coredumps in this scenario at all because the default kernel
configuration will only overcommit 50% of available RAM.
Avoid this failure by using a realloc() loop.
See: http://lists.freedesktop.org/archives/systemd-devel/2013-April/010709.html
Session objects will now get the .session suffix, user objects the .user
suffix, nspawn containers the .nspawn suffix.
This also changes the user cgroups to be named after the numeric UID
rather than the username, since this allows us the parse these paths
standalone without requiring access to the cgroup file system.
This also changes the mapping of instanced units to cgroups. Instead of
mapping foo@bar.service to the cgroup path /user/foo@.service/bar we
will now map it to /user/foo@.service/foo@bar.service, in order to
ensure that all our objects are properly suffixed in the tree.
This reverts commit 4826f0b7b5.
Because statfs.t_type can be int on some architecures, we have to cast
the const magic to the type, otherwise the compiler warns about
signed/unsigned comparison, because the magic can be 32 bit unsigned.
statfs(2) man page is also wrong on some systems, because
f_type is not __SWORD_TYPE on some architecures.
The following program:
int main(int argc, char**argv)
{
struct statfs s;
statfs(argv[1], &s);
printf("sizeof(f_type) = %d\n", sizeof(s.f_type));
printf("sizeof(__SWORD_TYPE) = %d\n", sizeof(__SWORD_TYPE));
printf("sizeof(long) = %d\n", sizeof(long));
printf("sizeof(int) = %d\n", sizeof(int));
if (sizeof(s.f_type) == sizeof(int)) {
printf("f_type = 0x%x\n", s.f_type);
} else {
printf("f_type = 0x%lx\n", s.f_type);
}
return 0;
}
executed on s390x gives for a btrfs:
sizeof(f_type) = 4
sizeof(__SWORD_TYPE) = 8
sizeof(long) = 8
sizeof(int) = 4
f_type = 0x9123683e
