Category: Uncategorized

Entity Framework Core 3

I recently upgraded an .NET Core 2 project to .NET Core 3 and with that I also updated other packages, including EF Core. Now EF Core 3 has quite a list of breaking changes you can se them in the list below.

https://docs.microsoft.com/en-us/ef/core/what-is-new/ef-core-3.0/breaking-changes

The worst change for my application was one that is quit a bit down on the list and one that I did not see until I found the actual issue in my code.

Before 3.0, eagerly loading collection navigations via Include operators caused multiple queries to be generated on relational database, one for each related entity type.

Eager loading of related entities now happens in a single query

This does not sound to bad, nothing will stop working, but it will effect performance. If it effects performance depends on your query, some will go faster and some will go slower.

In my case I had an query that did not like this change at all. the query created an list of items. These items then had a bunch of data from other tables, created an 3d dimensional object. I Mocked up an example below on how the Linq Query looked like.

return context.Item
	.Select(i => new ResultItem()
	{
		ItemId   = i.ItemId,
		Siblings = i.parentNavigation.Items.Count(),
		Position = i.parentNavigation.Items.Count(p => p.Order < I.Order),
		tags = i.Tag.Select(t => new ResultTag {
			tag = t.tag
		})
	}).ToList();

In EF Core 2, this worked quite well. EF Core divided the code in two parts. First it did one query to fetch the items, count the siblings and position, and then it would do multiple queries to fetch the tags. So if I fetched 12 items. it would be 1 query to fetch the items and then 12 queries to fetch all the tags.

EF Core 3 however does one query, ONE BIG QUERY. In this case it resulted in some performance drop. In EF Core 2 it had a stable 250ms execution time, independent on how many tags each item had. In EF Core 3 it hovered around 650ms but could be slower if one item had more tags than usual.

Improvement 1

var items = context.Item
	.Select(i => new ResultItem()
	{
		ItemId   = i.ItemId,
                ParentId = i.ParentId,
                Order = i.Order
	}).ToList();

foreach (var item in items)
{
	item.Siblings = context.Item.Count(i => i.ParentId == item.ParentId);
	item.Position = context.Item.Count(i => i.ParentId == item.ParentId && i.Order < item.Order);
	item.tags = context.Tag.Where(t => t.ItemId = item.ItemId).Select(t => new ResultTag {
			tag = t.tag
		}).ToList();
}

First step was to separate the different queries. We simply fetched the item first and then looped over them to fetch the extra data needed. This resulted in quite some improvement, we decreased the time from around 650ms to 350ms. But it was still not fast enough.

The reason it still not as fast as before is because this code results in a lot of queries. If we fetch 12 items, it will result 1 query to fetch the items, then it will loop over all the items and do 3 separate queries for each item. Resulting in a total of 1+36 queries against the database.

Each query means another roundtrip to the database. And this is partly the reason why EF Core decided to make the change in EF core 3, to try to avoid as many roundtrips as possible.

Improvement 2

var items = context.Item
	.Select(i => new ResultItem()
	{
		ItemId   = i.ItemId,
                ParentId = i.ParentId,
                Order = i.Order
	}).ToList();

var itemIds = items.Select(c => c.ItemId);
var tags = context.Tag.Where(t => itemIds.Contains(t.ItemId)).Select(t => new ResultTag {
	ItemId = t.ItemId,
	Tag = t.Tag
}).ToList();

var parentIds = items.Select(i => i.ParentId);
var siblings = context.Item.Where(i => parentIds.Contains(i.parentId)).Select(i => new SiblingItem {
	ParentId = i.ParentId,
	Order = i.Order
}).ToList();

foreach (var item in items)
{
	item.Siblings = siblings.Count(i => i.ParentId == item.ParentItem);
	item.Position = siblings.Count(i => i.ParentId == item.ParentItem && i.Order < item.Order);
	item.tags = tags.Where(t => t.ItemId = item.ItemId);
}

Now we have complicated our code somewhat, but this was the fastest way of doing it. As previous example we first fetch all the items. Our next step is for all items we fetched, select their siblings in on query and select all their tags in another query. We can then loop over the items and filter out the result for each item. This resulted in a total of 3 queries being sent to the database and the execution time dropped to under 100ms.

Podman/Docker cant reach internet

Had problem on CentOS 7/8 with both podman and docker. no container could reach internet. needed to enable the following.

cat <<EOF > /etc/sysctl.d/podman.conf
# Enable containers to access the outer world
net.ipv4.ip_forward=1
EOF

I also needed to enable masquerading on the external zone.

firewall-cmd --zone=public --add-masquerade
firewall-cmd --permanent --zone=public --add-masquerade

One question left, who is allowed to masquerade? i see no filtering done meaning any other network connected to this machine can masquerade as the public zone. Does not sound good.

icinga/nagios test icecast stream

This is howto monitor an icecast stream that the stream has not gone quiet and to do in during a specific time. First we need a tool to monitor the audio level. This can actually be done with ffmpeg. Below is a command you can do.

ffmpeg -t 10 -i http://example.com/live.mp3 -af "volumedetect" -f null /dev/null 2>&1 | grep Parsed_volumedetect

We do several things here.

  • -t 10 lets us just play the stream we have choosen for 10 seconds before closing ffmpeg
  • -i http://example.com/live.mp3 is our icecast stream we want to monitor
  • -af “volumedetect” is the audio filter we want to apply on the stream.
  • -f null /dev/null is to tell ffmpeg to throw the result away.
  • 2>&1 is very important, nomaly the output from ffmpeg you see on your screen is from stderr, by doing this in the end, we force the output out on the normal stdout giving us the possibilty to pipe the output to our grep command.
  • Lastly  grep Parsed_volumedetect is to only show us the output from the volumedetect filter.

Below we can see the date you can get from volumedetect. The value we want to use is mean_volume which will give us an aproximation on the current audio level in the stream. Max value is zero and anything below -40 db will be considered quite a low volume.

Now is the part where we convert the code above to a icinga/nagios test. I have uploaded a working example using the above code on https://github.com/Dalesjo/dalesjo-nagios/blob/master/media/check_audio_level Below you can se it in action, giving the correct exit code for icinga/nagios of course.

Continue reading

Icinga/Nagios test using zonemaster.

Zonemaster is a great tool to verify that you have set up your domainservers correctly. You can test it out on https://zonemaster.iis.se/en/

I want my icinga server to this automaticly so i will get a warning as soon something changes, so lets do that. First thing you need to now is that Zonemaster is a tool and is freely available on Github. you can download it and run it on your own machine.
Continue reading

raid0 for varnish

To speed up varnish i choosen to cache data to a mdadm raid0 partition on half the drives (rest is raid1).

Create raid0 partitions (done twice)

(parted) print
Model: ATA ST2000NC001-1DY1 (scsi)
Disk /dev/sda: 2000GB
Sector size (logical/physical): 512B/4096B
Partition Table: msdos
Disk Flags:
Number Start End Size Type File system Flags
1 1049kB 1001MB 1000MB primary boot, raid
2 1001MB 1009GB 1008GB primary raid

(parted) mkpart
Partition type? primary/extended? primary
File system type? [ext2]? xfs
Start? 1100GB
End? 1900GB

(parted) set 3 raid on
(parted) print
Model: ATA ST2000NC001-1DY1 (scsi)
Disk /dev/sda: 2000GB
Sector size (logical/physical): 512B/4096B
Partition Table: msdos
Disk Flags:

Number Start End Size Type File system Flags
1 1049kB 1001MB 1000MB primary boot, raid
2 1001MB 1009GB 1008GB primary raid
3 1100GB 1900GB 800GB primary raid

mdadm raid0

mdadm -C /dev/md0 -l raid0 -n 2 /dev/sd[a-b]3
mkfs.xfs /dev/md0
mdadm --detail --scan >> /etc/mdadm.conf

fstab

To make sure this server can start evan if the raid0 partition fails, add nofail. noatime is added since atime is not needed for this drive.

/dev/md0  /var/lib/varnish/                     xfs     defaults,nofail,noatime 0 0

If it fails

If the drive fails varnish will crash, at a reboot varnish will not start (this is because the file varnish want to use dont fit within the root partition. to fix the raid0 partition, recreate all steps it and mount. dont forget to update /etc/mdadm.conf.