Updated: 6:10 p.m. ET.

The Pentagon is pausing development of Advanaits default data-analytics platformso it can be upgraded to handle increased demand, according to an internal email obtained by Defense One.

In the June 3 email, the Pentagons new chief data and artificial intelligence officerdirected developers of the Advana platform to pause much of the active work and additional features until infrastructure changes are complete.

The pause will force users who were banking on forthcomingfeatures, tools, or applications to use existing Advana tools to do their work. The email did not indicate when development might resume.

The Department's demand for enterprise data and analytics services have outgrown the original architecture of the Advana platform, wrote Radha Iyengar Plumb, reflecting a review of the infrastructure, tech tools, and applications that her office relies on soon after taking office in April.

I have directed the team to focus on upgrading the technical framework of Advana to better meet the requirements of our customers and the whole Department, and develop a sustainable enterprise solution for the future, Plumb wrote. To accelerate these platform enhancements, I have also directed the team to reprioritize activities from continuing to build on Advana's current platform to focus on its future platform engineering activities. As part of this, we are looking hard at a variety of applications in the Advana ecosystem.

During the developmental pause, the Pentagon will evaluate whether the applications are stable and should be integrated on the upgraded infrastructure, or if they would fit better elsewhere in DODs tech enterprise.

I understand this will disrupt the planned uses and services for your teams, and I did not make this decision lightly. I understand from my team that this reverses prior commitments madeand want to acknowledge the impacts this may have on your roadmaps and leadership obligations, Plumb wrote. My team and l are dedicated to working with you all to identify alternative hosting environments for your use case or transition your use case to generally available tools on [Advana] until we are able to relook at bringing new vendor pilots onto the future infrastructure environment.

The Advana platform has been an important part of the Pentagons data and analytics efforts, getting its start with financial data management and growing to include other areas across the defense enterprise. In a 2021 memo, Deputy Defense Secretary Kathleen Hicks billed Advana as the Pentagons go-to data analytics platform, and said defense organizations must get executive approval to use other platforms.

The Advancing Analytics (Advana) platform is the single enterprise authoritative data management and analytics platform for the Secretary of Defense, Deputy Secretary of Defense, and Principal Staff Assistants (PSAs), with inputs from all DoD Components, Hicks wrote. The use of other data management and analytics platforms must be approved by the DoD CDO and appropriate Component CDO to ensure adherence to an open data standard architecture.

Researchers have suggested that unified data analytics platforms that tie disparate systems together, like Advana, could help lessen the workloads of Pentagon employees. Moreover, Advana is the authoritative source for reporting Ukraine supplemental funding, and has been praised for its ability to keep track of weapons shipmentsbut criticized for the added burden to soldiers.

Having a mission-ready enterprise analytics infrastructure is critical to the Departments goal of leveraging data and AI from the boardroom to the battlefield, a senior defense official told Defense One.

In May, the Pentagon announced a new initiative, called the Open Data and Applications Government-owned Interoperable Repositories framework, designed to bring data analytics across the defense enterprise. The goal is to build on Hicks 2021 data memo to create a multi-vendor analytics ecosystem.

Advana is an important part of that data infrastructure layer and application environment. Over the past two years, the Advana platform scaled rapidly from initial capability based on pilot projects and prototypes to an enterprise-wide data and analytics environment the Department now uses to inform decision making at all levels, the official said, adding that the planned upgrades will help support Advanas rapid user growth.

These changes will result in improvements, such as accelerating the onboarding of new use cases for enterprise customers and opening up the Advana data platform to third party software development at a larger scale than we enjoy today. Our work to upgrade the platform will impact back-end engineering and, for the most part, no Advana customers will experience a degradation of existing services.

Read this article:

Pentagon pauses development of its go-to data analytics tool - Defense One

Because we are using an unsupervised learning algorithm, there is not a widely available measure of accuracy. However, we can use domain knowledge to validate our groups.

Visually inspecting the groups, we can see some benchmarking groups have a mix of Economy and Luxury hotels, which doesn't make business sense as the demand for hotels is fundamentally different.

We can scroll to the data and note some of those differences, but can we come up with our own accuracy measure?

We want to create a function to measure the consistency of the recommended Benchmarking sets across each feature. One way of doing this is by calculating the variance in each feature for each set. For each cluster, we can compute an average of each feature variance, and we can then average each hotel cluster variance to get a total model score.

From our domain knowledge, we know that in order to set up a comparable benchmark set, we need to prioritize hotels in the same Brand, possibly the same market, and the same country, and if we use different markets or countries, then the market tier should be the same.

With that in mind, we want our measure to have a higher penalty for variance in those features. To do so, we will use a weighted average to calculate each benchmark set variance. We will also print the variance of the key features and secondary features separately.

To sum up, to create our accuracy measure, we need to:

To keep our code clean and track our experiments , lets also define a function to store the results of our experiments.

Now that we have a baseline, lets see if we can improve our model.

Up until now, we did not have to know what was going on under the hood when we ran this code:

To improve our model, we will need to understand the model parameters and how we can interact with them to get better benchmark sets.

Lets start by looking at the Scikit Learn documentation and source code:

There are quite a few things going on here.

The Nearestneighbor class inherits fromNeighborsBase, which is the case class for nearest neighbor estimators. This class handles the common functionalities required for nearest-neighbor searches, such as

The Nearestneighbor class also inherits fromKNeighborsMixin and RadiusNeighborsMixinclasses. These Mixin classes add specific neighbor-search functionalities to the Nearestneighbor

Based on our scenario, KNeighborsMixin provides the functionality we need.

We need to understand one key parameter before we can improve our model; this is the distance metric.

The documentation mentions that the NearestNeighbor algorithm uses the Minkowski distance by default and gives us a reference to the SciPy API.

In scipy.spatial.distance, we can see two mathematical representations of "Minkowski" distance:

uv p=( i u iv i p ) 1/p

This formula calculates the p-th root of the sum of powered differences across all elements.

The second mathematical representation of Minkowski distance is:

uv p=( i w i(u iv i p )) 1/p

This is very similar to the first one, but it introduces weights wi to the differences, emphasizing or de-emphasizing specific dimensions. This is useful where certain features are more relevant than others. By default, the setting is None, which gives all features the same weight of 1.0.

This is a great option for improving our model as it allows us to pass domain knowledge to our model and emphasize similarities that are most relevant to users.

If we look at the formulas, we see the parameter. p. This parameter affects the "path" the algorithm takes to calculate the distance. By default, p=2, which represents the Euclidian distance.

You can think of the Euclidian distance as calculating the distance by drawing a straight line between 2 points. This is usally the shortest distance, however, this is not always the most desirable way of calculating the distance, specially in higher dimention spaces. For more information on why this is the case, there is this great paper online: https://bib.dbvis.de/uploadedFiles/155.pdf

Another common value for p is 1. This represents the Manhattan distance. You think of it as the distance between two points measured along a grid-like path.

On the other hand, if we increase p towards infinity, we end up with the Chebyshev distance, defined as the maximum absolute difference between any corresponding elements of the vectors. It essentially measures the worst-case difference, making it useful in scenarios where you want to ensure that no single feature varies too much.

By reading and getting familiar with the documentation, we have uncovered a few possible options to improve our model.

By default n_neighbors is 5, however, for our benchmark set, we want to compare each hotel to the 3 most similar hotels. To do so, we need to set n_neighbors = 4 (Subject hotel + 3 peers)

Based on the documentation, we can pass weights to the distance calculation to emphasize the relationship across some features. Based on our domain knowledge, we have identified the features we want to emphasize, in this case, Brand, Market, Country, and Market Tier.

Passing domain knowledge to the model via weights increased the score significantly. Next, lets test the impact of the distance measure.

So far, we have been using the Euclidian distance. Lets see what happens if we use the Manhattan distance instead.

Decreasing p to 1 resulted in some good improvements. Lets see what happens as p approximates infinity.

To use the Chebyshev distance, we will change the metric parameter to Chebyshev. The default sklearn Chebyshev metric doesnt have a weight parameter. To get around this, we will define a custom weighted_chebyshev metric.

We managed to decrease the primary feature variance scores through experimentation.

Lets visualize the results.

Using Manhattan distance with weights seems to give the most accurate benchmark sets according to our needs.

The last step before implementing the benchmark sets would be to examine the sets with the highest Primary features scores and identify what steps to take with them.

These 18 cases will need to be reviewed to ensure the benchmark sets are relevant.

As you can see, with a few lines of code and some understanding of Nearest neighbor search, we managed to set internal benchmark sets. We can now distribute the sets and start measuring hotels' KPIs against their benchmark sets.

You dont always have to focus on the most cutting-edge machine learning methods to deliver value. Very often, simple machine learning can deliver great value.

What are some low-hanging fruits in your business that you could easily tackle with Machine learning?

World Bank. World Development Indicators. Retrieved June 11, 2024, from https://datacatalog.worldbank.org/search/dataset/0038117

Aggarwal, C. C., Hinneburg, A., & Keim, D. A. (n.d.). On the Surprising Behavior of Distance Metrics in High Dimensional Space. IBM T. J. Watson Research Center and Institute of Computer Science, University of Halle. Retrieved from https://bib.dbvis.de/uploadedFiles/155.pdf

SciPy v1.10.1 Manual. scipy.spatial.distance.minkowski. Retrieved June 11, 2024, from https://docs.scipy.org/doc/scipy/reference/generated/scipy.spatial.distance.minkowski.html

GeeksforGeeks. Haversine formula to find distance between two points on a sphere. Retrieved June 11, 2024, from https://www.geeksforgeeks.org/haversine-formula-to-find-distance-between-two-points-on-a-sphere/

scikit-learn. Neighbors Module. Retrieved June 11, 2024, from https://scikit-learn.org/stable/modules/classes.html#module-sklearn.neighbors

See the original post:

Improving Business Performance with Machine Learning | by Juan Jose Munoz | Jun, 2024 - Towards Data Science

Soham Jariwala has a unique perspective: He took the Hackathon course in 2022 and this spring is a project mentor alongside Vasu Venkateshwaran from W. L. Gore. & Associates. Jariwala, a doctoral alumnus from the chemical and biomolecular engineering department and now a modeling and simulation scientist at Gore, was not officially part of the NRT program but took the course to gain hands-on experience with using machine learning tools for industry projects, an experience that he said was formative in helping him succeed in his current role.

In a traditional classroom, you have a limited perspective on how projects are conducted in industry, Jariwala said. In the Hackathon class, you have a problem that even the industry experts don't know the answer to. As a team, you bring your own expertise, brainstorm ideas, find the best approach, and learn about other areas in order to reach a decision.

The summer after completing the Hackathon course, students in the NRT-MIDAS program have the option of either completing a summer internship or a two-week teaching workshop.

Alison Shapiro, a chemical engineering doctoral candidate working in the lab of Allan & Myra Ferguson Distinguished ProfessorThomas H. Epps, III, worked at Dow last summer as part of the companys cable and wire department. During her time at Dow, she looked for ways to recycle and more sustainably fabricate the insulating, protective polymers that coat electrical wires and conducted life cycle assessments for candidate materials.

Shapiro said that understanding the vernacular used to talk about chemicals during the Hackathon course was extremely helpful in completing her internship projects.

At Dow, they had the same way of talking about [formulations] as we did in the Hackathon class, which is different from how most academic research was done, she said. That was one of the things that translated over the most, and I initially had no idea that it was going to be so helpful.

Sean Farrington, who is also part of the first NRT cohort, is a doctoral candidate working under Unidel Robert L. Pigford ChairNorman Wagner and Arthur B. Metzner ProfessorAntony Beris. Last summer he completed a teaching workshop developed by NRT core faculty member and associate professorJoshua Enszer, which involved presentations about class preparation and teaching strategies followed by each student delivering a mock lecture and receiving constructive feedback.

Farrington, who is currently a TA in the chemical engineering department, regularly uses a list of action verbs provided by Enszer when preparing to teach. He said that the workshop was invaluable, not only for his current career plans of working in academia but because no matter what job you have, you are always going to have to teach people something, and to do so you need to figure out exactly what your learning outcomes are, he said.

Outside of the programs coursework and professional development activities, NRT-MIDAS also fosters a strong sense of community that reaches across multiple departments.

This includes a biweekly NRT community hour organized by Johnston and Jayaraman, where all members of the NRT-MIDAS community get together during the lunch hour. Along with socializing over pizza, students get to hear invited speakers discuss their research in academic and national laboratories, learn about various STEM careers in industry, publishing, and teaching, and attend professional development workshops on topics such as data ethics, responsible conduct of research, and science communication.

As NRT program coordinator, Johnston plays a key role in helping foster this sense of community, from helping students become comfortable with public speaking and communication during their outreach activities to hosting monthly individual advising meetings with all of the trainees, which Johnston said is the highlight of her week.

Not only do I enjoy getting to know our students, but they also provide valuable information on what they need from us as a program, Johnston said. These meetings have helped influence our professional skill community hour topics and have given us the ability to really cater to the needs of the students in our program.

During the summer, students work in teams to complete an outreach activity that showcases STEM research and data science concepts for a variety of non-scientific audiences.

In the summer of 2022, the first cohort createdvideos to help explain their research and pique other students interest in science.

Read more from the original source:

Computers and chemistry | UDaily - University of Delaware

15 min read

Im excited to share my journey of building a unique Retrieval-Augmented Generation (RAG) application for interacting with rabbinic texts in this post. MishnahBot aims to provide scholars and everyday users with an intuitive way to query and explore the Mishnah interactively. It can help solve problems such as quickly locating relevant source texts or summarizing a complex debate about religious law, extracting the bottom line.

I had the idea for such a project a few years back, but I felt like the technology wasnt ripe yet. Now, with advancements of large language models, and RAG capabilities, it is pretty straightforward.

This is what our final product will look like, which you could try out here:

RAG applications are gaining significant attention, for improving accuracy and harnessing the reasoning power available in large language models (LLMs). Imagine being able to chat with your library, a collection of car manuals from the same manufacturer, or your tax documents. You can ask questions, and receive answers informed by the wealth of specialized knowledge.

There are two emerging trends in improving language model interactions: Retrieval-Augmented Generation (RAG) and increasing context length, potentially by allowing very long documents as attachments.

One key advantage of RAG systems is cost-efficiency. With RAG, you can handle large contexts without drastically increasing the query cost, which can become expensive. Additionally, RAG is more modular, allowing you to plug and play with different knowledge bases and LLM providers. On the other hand, increasing the context length directly in language models is an exciting development that can enable handling much longer texts in a single interaction.

For this project, I used AWS SageMaker for my development environment, AWS Bedrock to access various LLMs, and the LangChain framework to manage the pipeline. Both AWS services are user-friendly and charge only for the resources used, soIreallyencourageyoutotryitoutyourselves. For Bedrock, youll need to request access to Llama 3 70b Instruct and Claude Sonnet.

Lets open a new Jupyter notebook, and install the packages we will be using:

The dataset for this project is the Mishnah, an ancient Rabbinic text central to Jewish tradition. I chose this text because it is close to my heart and also presents a challenge for language models since it is a niche topic. The dataset was obtained from the Sefaria-Export repository, a treasure trove of rabbinic texts with English translations aligned with the original Hebrew. This alignment facilitates switching between languages in different steps of our RAG application.

Note: The same process applied here can be applied to any other collection of texts of your choosing. This example also demonstrates how RAG technology can be utilized across different languages, as shown with Hebrew in this case.

First we will need to download the relevant data. We will use git sparse-checkout since the full repository is quite large. Open the terminal window and run the following.

And voila! we now have the data files that we need:

Now lets load the documents in our Jupyter notebook environment:

And take a look at the Data:

Looks good, we can move on to the vector database stage.

Next, we vectorize the text and store it in a local ChromaDB. In one sentence, the idea is to represent text as dense vectors arrays of numbers such that texts that are similar semantically will be close to each other in vector space. This is the technology that will enable us to retrieve the relevant passages given a query.

We opted for a lightweight vectorization model, the all-MiniLM-L6-v2, which can run efficiently on a CPU. This model provides a good balance between performance and resource efficiency, making it suitable for our application. While state-of-the-art models like OpenAIs text-embedding-3-large may offer superior performance, they require substantial computational resources, typically running on GPUs.

For more information about embedding models and their performance, you can refer to the MTEB leaderboard which compares various text embedding models on multiple tasks.

Heres the code we will use for vectorizing (should only take a few minutes to run on this dataset on a CPU machine):

With our dataset ready, we can now create our Retrieval-Augmented Generation (RAG) application in English. For this, well use LangChain, a powerful framework that provides a unified interface for various language model operations and integrations, making it easy to build sophisticated applications.

LangChain simplifies the process of integrating different components like language models (LLMs), retrievers, and vector stores. By using LangChain, we can focus on the high-level logic of our application without worrying about the underlying complexities of each component.

Heres the code to set up our RAG system:

Alright! Lets try it out! We will use a query related to the very first paragraphs in the Mishnah.

That seems pretty accurate.

Lets try a more sophisticated question:

Very nice.

I tried that out, heres what I got:

The response is long and not to the point, and the answer that is given is incorrect (reaping is the third type of work in the list, while selecting is the seventh). This is what we call a hallucination.

While Claude is a powerful language model, relying solely on an LLM for generating responses from memorized training data or even using internet searches lacks the precision and control offered by a custom database in a Retrieval-Augmented Generation (RAG) application. Heres why:

This structured retrieval process ensures users receive the most accurate and relevant answers, leveraging both the language generation capabilities of LLMs and the precision of custom data retrieval.

Finally, we will address the challenge of interacting in Hebrew with the original Hebrew text. The same approach can be applied to any other language, as long as you are able to translate the texts to Englishfortheretrievalstage.

Supporting Hebrew interactions adds an extra layer of complexity since embedding models and large language models (LLMs) tend to be stronger in English. While some embedding models and LLMs do support Hebrew, they are often less robust than their English counterparts, especially the smaller embedding models that likely focused more on English during training.

To tackle this, we could train our own Hebrew embedding model. However, another practical approach is to leverage a one-time translation of the text to English and use English embeddings for the retrieval process. This way, we benefit from the strong performance of English models while still supporting Hebrew interactions.

In our case, we already have professional human translations of the Mishnah text into English. We will use this to ensure accurate retrievals while maintaining the integrity of the Hebrew responses. Heres how we can set up this cross-lingual RAG system:

For generation, we use Claude Sonnet since it performs significantly better on Hebrew text compared to Llama 3.

Here is the code implementation:

Lets try it! We will use the same question as before, but in Hebrew this time:

We got an accurate, one word answer to our question. Pretty neat, right?

The translation with Llama 3 Instruct posed several challenges. Initially, the model produced nonsensical results no matter what I tried. (Apparently, Llama 3 instruct is very sensitive to prompts starting with a new line character!)

After resolving that issue, the model tended to output the correct response, but then continue with additional irrelevant text, so stopping the output at a newline character proved effective.

Controlling the output format can be tricky. Some strategies include requesting a JSON format or providing examples with few-shot prompts.

In this project, we also remove vowels from the Hebrew texts since most Hebrew text online does not include vowels, and we want the context for our LLM to be similar to text seen during pretraining.

Building this RAG application has been a fascinating journey, blending the nuances of ancient texts with modern AI technologies. My passion for making the library of ancient rabbinic texts more accessible to everyone (myself included) has driven this project. This technology enables chatting with your library, searching for sources based on ideas, and much more. The approach used here can be applied to other treasured collections of texts, opening up new possibilities for accessing and exploring historical and cultural knowledge.

Its amazing to see how all this can be accomplished in just a few hours, thanks to the powerful tools and frameworks available today. Feel free to check out the full code on GitHub, and play with the MishnahBot website.

Please share your comments and questions, especially if youre trying out something similar. If you want to see more content like this in the future, do let me know!

Follow this link:

Exploring RAG Applications Across Languages: Conversing with the Mishnah - Towards Data Science

A recent release of Julia such as 1.10 is recommended. For those wanting to use a notebook, the repository shared above also contains a Pluto file, for which Pluto.jl needs to be installed. The input data file for the challenge is unique for everyone and needs to be generated using this Python script. Keep in mind that the file is about 15 GB in size.

Additionally, we will be running benchmarks using the BenchmarkTools.jl package. Note that this does not impact the challenge, its only meant to collect proper statistics to measure and quantify the performance of the Julia code.

The structure of the input data file measurements.txt is as follows (only the first five lines are shown):

The file contains a billion lines (also known as rows or records). Each line has a station name followed by the ; separator and then the recorded temperature. The number of unique stations can be up to 10,000. This implies that the same station appears on multiple lines. We therefore need to collect all the temperatures for all distinct stations in the file, and then calculate the required statistics. Easy, right?

My first attempt was to simply parse the file one line at a time, and then collect the results in a dictionary where every station name is a key and the temperatures are added to a vector of Float64 to be used as the value mapped to the key. I expected this to be slow, but our aim here is to get a number for the baseline performance.

Once the dictionary is ready, we can calculate the necessary statistics:

The output of all the data processing needs to be displayed in a certain format. This is achieved by the following function:

Since this implementation is expected to take long, we can run a simple test by timing @time the following only once:

Our poor mans implementation takes about 526 seconds, so ~ 9 minutes. Its definitely slow, but not that bad at all!

Instead of reading the input file one line at a time, we can try to split it into chunks, and then process all the chunks in parallel. Julia makes it quite easy to implement a parallel for loop. However, we need to take some precautions while doing so.

Before we get to the loop, we first need to figure out how to split the file into chunks. This can be achieved using memory mapping to read the file. Then we need to determine the start and end positions of each chunk. Its important to note that each line in the input data file ends with a new-line character, which has 0x0a as the byte representation. So each chunk should end at that character to ensure that we dont make any errors while parsing the file.

The following function takes the number of chunksnum_chunksas an input argument, then returns an array with each element as the memory mapped chunk.

Since we are parsing station and temperature data from different chunks, we also need to combine them in the end. Each chunk will first be processed into a dictionary as shown before. Then, we combine all chunks as follows:

Now we know how to split the file into chunks, and how we can combine the parsed dictionaries from the chunks at the end. However, the desired speedup can only be obtained if we are also able to process the chunks in parallel. This can be done in a for loop. Note that Julia should be started with multiple threads julia -t 12 for this solution to have any impact.

Additionally, we now want to run a proper statistical benchmark. This means that the challenge should be executed a certain number of times, and we should then be able to visualize the distribution of the results. Thankfully, all of this can be easily done with BenchmarkTools.jl. We cap the maximum number of samples to 10, maximum time for the total run to be 20 minutes and enable garbage collection (will free up memory) to execute between samples. All of this can be brought together in a single script. Note that the input arguments are now the name of the file fname and the number of chunks num_chunks.

Benchmark results along with the inputs used are shown below. Note that we have used 12 threads here.

Multi-threading provides a big performance boost, we are now down to roughly over 2 minutes. Lets see what else we can improve.

Until now, our approach has been to store all the temperatures, and then determine the required statistics (min, mean and max) at the very end. However, the same can already be achieved while we parse every line from the input file. We replace existing values each time a new value which is either larger (for maximum) or smaller (for minimum) is found. For mean, we sum all the values and keep a separate counter as to how many times a temperature for a given station has been found.

Overall, out new logic looks like the following:

The function to combine all the results (from different chunks) also needs to be updated accordingly.

Lets run a new benchmark and see if this change improves the timing.

The median time seems to have improved, but only slightly. Its a win, nonetheless!

Our previous logic to calculate and save the mix, max for temperature can be further simplified. Moreover, following the suggestion from this Julia Discourse post, we can make use of views (using @view ) when parsing the station names and temperature data. This has also been discussed in the Julia performance manual. Since we are using a slice expression for parsing every line, @view helps us avoid the cost of allocation and copying.

Rest of the logic remains the same. Running the benchmark now gives the following:

Whoa! We managed to reach down to almost a minute. It seems switching to a view does make a big difference. Perhaps, there are further tweaks that could be made to improve performance even further. In case you have any suggestions, do let me know in the comments.

Restricting ourselves only to base Julia was fun. However, in the real world, we will almost always be using packages and thus making use of existing efficient implementations for performing the relevant tasks. In our case, CSV.jl (parsing the file in parallel) and DataFrames.jl (performing groupby and combine) will come in handy.

The function below performs the following tasks:

We can now run the benchmark in the same manner as before.

The performance using CSV.jl and DataFrames.jl is quite good, albeit slower than our base Julia implementation. When working on real world projects, these packages are an essential part of a data scientists toolkit. It would thus be interesting to explore if further optimizations are possible using this approach.

See more here:

The One Billion Row Challenge in Julia | by Vikas Negi | Jun, 2024 - Towards Data Science