Core Mio
This article will explore the thread-per-core model with its advantages and challenges, and introduce Glommio (you can also find it on crates.io), our solution to this problem. Glommio allows Rust developers to write thread-per-core applications in an easy and manageable way.
Core mio
We know that thread-per-core can deliver significant efficiency gains. But what is it? In simple terms, any moderately complex application has many tasks that it needs to perform: it may need to read data from a database, feed that data through a machine learning model, and then pass that result along the pipeline. Some of those tasks are naturally sequential, but many can be done in parallel. And since modern hardware keeps increasing the number of cores available for applications, it is important to efficiently use them to achieve good performance numbers.
Thread-per-core programming eliminates threads from the picture altogether. Each core, or CPU, runs a single thread, and often (although not necessarily), each of these threads is pinned to a specific CPU. As the Operating System Scheduler cannot move these threads around, and there is never another thread in that same CPU, there are no context switches.
To take advantage of thread-per-core, developers should employ sharding: each of the threads in the thread-per-core application becomes responsible for a subset of the data. For example, it could be that each thread will read from a different Kafka partition, or that each thread is responsible for a subset of the keys in a database. Anything is possible, so long as two threads never share the responsibility of handling a particular request. As scalability concerns become the norm rather than the exception, sharding is usually already present in modern applications in one form or another: thread-per-core, in this case, becomes the cherry on top.
The thread-per-core design takes this one step further: we know that updates to Key 3 and Key 4 are serialized. They have to be! If they run in the same thread, then we are either operating on Key 3 or Key 4, never both. So long as we finish the update before declaring the task complete, the locks are gone. As we can see in the figure below, all possible update tasks for each of the cache shards are naturally serialized, and only one (in purple) runs at a time. So as long as it finishes its update before leaving the thread, locks are not necessary.
I wish! Thread-per-core has been around for a while. As a matter of fact, for many years before I joined Datadog, I worked in a thread-per-core framework for C++ called Seastar, the engine that is behind the ScyllaDB NoSQL database. ScyllaDB managed to leverage the thread-per-core model to provide more efficient implementations of existing databases like Apache Cassandra, so I knew that the model would work for our datastores too while keeping the complexity manageable.
Linux is ubiquitous in the modern datacenter, to the point that we can take advantage of Linux-only APIs like io_uring to bring things like Glommio to fruition. But another technology that is slowly but surely reaching that status is Kubernetes. Kubernetes is a flexible abstraction, where pods can be running everywhere. That begs the question: will a thread-per-core architecture do well on Kubernetes?
The answer is yes: thread-per-core applications will run on any Kubernetes infrastructure. However, best performance will come from matching the application to the physical cores available in the underlying hardware. To do that effectively:
As hardware gets faster and more feature rich, it is important to bring applications in line with new techniques to take full advantage of what the hardware provides. Modern applications that need to be sharded for scalability are prime candidates for using a thread-per-core architecture, where each CPU will have sole control over a fragment of the dataset.
Thread-per-core architectures are friendly to modern hardware, as their local nature helps the application to take advantage of the fact that processors ship with more and more cores while storage gets faster, with modern NVMe devices having response times in the ballpark of an operating system context switch.
Washing: Wash before use. Machine wash with enzyme-free detergent up to 140F. Do not use fabric softener. Do not use chlorine bleach. If soaking, only soak the diaper core, keeping the laminated outer fabric (shell) out of the water. To add a boost to your laundry, add a dash of laundry treatment to your laundry. Please note: colors may fade over time. Drying: Air dry away from direct heat. Can be tumble dried on a cold/low setting. Do not dry clean. Do not iron. TIP: Be kind to your diapers - to get the most out of them, wash at lower temperatures. Washing at higher temperatures and frequent tumble drying can result in premature deterioration of your diapers. Download our laundry guide.
So I have seen this previously too - This query did a pull from snowflake in about 18 mins and the majority of the rest of the time (before it failed with lack of disk space), it chugged away with just one core processing - This has the effect of making it much slower than snowflake.
The reason we are executing on only one core is because you have only one reflection Parquet file and in that only one group which decides the number of splits which in turn decides the number of parallel threads. Number of rowgroups upto 70% of the total number of cores on that node will decide the degree of parallelism. I see you query is accelerated so when you create the reflection you need to decide on the right partition column so you get more splits, below is a best practices document for creating reflections
Long (>100ka) records of climatic change from terrestrial environments in Siberia are rare but essential to improve our understanding of the Arctic's role in global climate dynamics. North-east Siberia provides a key area to study climatic teleconnections between the North Pacific oceanic system, climatic pattern over NE Russia, the Arctic Ocean and other climate forcing areas such as the North Atlantic and the Tropics. Lake El'gygytgyn, located in central Chukotka, NE Russia, was formed 3.6 million years ago by a meteorite impact and apparently escaped continental scale glaciations during the entire Quaternary. If so a full-length sediment core would yield a complete record of Arctic climate evolution, back one million years prior to the first major glaciation of the Northern Hemisphere. A 13.0 m long sediment core retrieved from the lake in 1998 revealed a basal age of approx. 250 ka, confirmed the lack of glacial erosion, and underlined the sensitivity of this lacustrine environment to reflect high resolution climatic change on Milankovitch and sub-Milankovitch time scales. Seismic investigation carried out during two expedtions in 2000 and 2003 revealed a depth-velocity model of brecciated bedrock overlain by a suevite layer overlain by two lacustrine sedimentary units up to 350 m in thickness. The upper well-stratified sediment unit appears undisturbed apart from intercalation with debris flows near the slopes. Based on extrapolation of sedimentation rates the entire Quaternary and possibly parts of the late Tertiary record are within the 170m thick unit one and the earliest history of the lake is in unit two. There is no evidence of glacial erosion in the sedimentary record. High-resolution 3.5 kHz profiles indicate sharp termination lobes of non-erosive debris flows in distal areas. Near the centre of the lake the 250ka sediment-core record exhibit a few thin distal turbidites possibly generated by debris flows. The character of the sediment fill suggests a high potential of the record for paleoclimate studies and deep drilling would offers opportunities of impact studies of the brecciated bedrock. Our study is part of international and multi-disciplinary site-survey investigation of Lake El'gygytgyn. The lake has been recognised as potential deep drilling location by the International Continental Drilling Program (ICDP).
A continuous hemipelagic sedimentary section reaching the middle Miocene (15 Ma) was recovered from Site 1085. The micropaleontological studies were carried out on core-catcher samples from Hole 1085A. Additional samples from within the cores were examined to improve the biostratigraphic resolution. A high-resolution biostratigraphy was developed using calcareous nannofossils and planktonic foraminifers. Sedimentation rates range from 1.5 to 13 cm/k.y. The lowest sedimentation rates are within the middle Mio-cene (1.5 cm/k.y.) and the highest are within the Pleistocene (13 cm/k.y.). Two other intervals with high sedimentation rates occur within the early part of the late Pliocene (7 cm/k.y.) and across the Miocene/Pliocene boundary (8 cm/k.y.).
Calcareous nannofossils were studied in core-catcher samples from Hole 1085A. Additional samples from within the top 11 cores (top 100 mbsf) were examined close to datum events to improve the stratigraphic resolution. Nannofossils are abundant and well preserved throughout the entire section. Reworking (trace; early Plio-cene specimen) was limited only to Cores 175-1085A-8H through 11H.
The uppermost assemblage (Sample 175-1085A-1H-CC) is dominated by Globigerina bulloides, Globorotalia inflata, and Neogloboquadrina pachyderma. Other species that are present include Globigerina quinqueloba, G. umbilicata, Globigerinella siphonifera, Globigerinoides ruber, G. sacculifer, G. crassaformis, G. hirsuta, G. scitula, G. truncatulinoides, N. dutertrei, and Orbulina universa. The presence of a warm-water species (including one endemic to the Indo-Pacific Ocean, G. hexagona) downcore (e.g., Sample 175-1085A-7H-CC) indicates transport of warm Indian Ocean water around the cape by the Agulhas Current.
The benthic foraminiferal fauna of Site 1085 was studied in selected core-catcher samples from Hole 1085A. The overall abundance of benthic foraminifers was high throughout the studied interval, except for the lowermost core catcher (Sample 175-1085A-64X-CC; 604.29 mbsf), which contained rare to few benthic foraminifers (Table 4). The planktonic to benthic ratio at this site is very high, about ten times higher than at previous sites. Preservation is good throughout Hole 1085A. 041b061a72