Technology peripherals
AI
Take stock of more than 20 powerful and free data sources that anyone can use to build AI
Take stock of more than 20 powerful and free data sources that anyone can use to build AI
When we talk about artificial intelligence in business and society today, we are actually referring to machine learning. Machine learning is an application that uses an algorithm (a set of instructions) to become better and better at performing a specific task as it is exposed to more and more data relevant to that task.
These tasks can be anything from answering questions, creating text or images (as apps like ChatGPT or Dall-E can do) to recognizing images (computer vision) or getting a self-driving car from point A to Navigate to location B.
Enterprises that want to train their own machine learning algorithms to automate daily tasks need data sources to support these tasks.
What types of data are there?
Enterprise data is usually divided into two categories - internal data and external data.
- Internal data is data collected by the enterprise organization itself from its operations, which usually includes financial data, customer feedback data, human resources data, operational data, and other data from more sources. Data collected by an organization in the course of monitoring its own operations is called proprietary data and is valuable because it provides information about a specific business.
- External data is data from sources external to the organization, typically collected from third-party data sources listed below. If the data is freely available to anyone, it is called open data.
In addition, data can also be divided into structured, unstructured or semi-structured data.
- Structured data is information that fits nicely and neatly into a table—for example, sales data showing what products a business sells, when, where, and for what price is internally structured data. Alternatively, businesses may choose to analyze historical market data and economic indicators to predict future trends in their markets (structured external data).
- Unstructured data is everything else, such as images, videos, text, and social media content, which can certainly contain valuable insights but is more difficult to analyze. However, AI has proven particularly useful for extracting meaning from unstructured data. For example, image recognition algorithms can tell businesses useful information about customer behavior by analyzing in-store CCTV images (internal unstructured data), and also by analyzing business-related images posted on social media (unstructured external data) to find valuable insights.
Fortunately, data is everywhere. Governments, research institutions, private companies, NGOs all provide data for free for research and even commercial purposes. So here are some of the best sources of free online data available in 2023.
Data Search Engine and Repository
- Google Dataset Search – This is essentially a search engine for Google’s cataloged datasets; use this search engine to find almost anything you might need data.
- AWS Open Data Search - Another data set search engine provided by Amazon's AWS.
- Microsoft Research Open Data - A free, open dataset collected by Microsoft with a primary focus on science.
- UCI Machine Learning Repository - A repository of more than 600 open datasets curated and maintained by the University of California, Irvine, that can be used to train machine learning algorithms.
- Kaggle Datasets – The online data science platform Kaggle also offers a curated catalog of datasets covering everything from university rankings to Google search trends, retail sales, online movie reviews and crime statistics.
- Reddit R/Datasets - Huge datasets submitted by users of the online community site Reddit, covering hundreds of topics.
Datasets for Governments and Intergovernmental Organizations
- Data.Gov - an open data portal provided by the U.S. government, hosting one million data published by government agencies Nearly a quarter of the data is concentrated.
- Data.Census.Gov – If you’re looking specifically for U.S. demographic data, this is a great place to start!
- Data.EU – The European Union’s open data portal, containing data from EU organizations and data from member state governments.
- Data.gov.uk - An open dataset published by UK government agencies.
- World Health Organization Data - Data sets related to global health and well-being.
- World Bank Open Data - Data sets related to economic development, international financial markets, social indicators and environmental issues.
Image Data
- Google Open Images - Millions of images classified and labeled in various ways, used to train many different types of computer vision algorithms .
- ImageNet Open Dataset - Another dataset consisting of labeled images that is free for non-commercial machine learning applications.
- COCO Dataset - The Common Objects in Context (COCO) dataset contains over 200,000 images selected for training object detection and captioning algorithms.
Voice Data
- Mozilla Common Voice - an open recording data set that can be used to train any AI application involving speech.
- Audioset - Another dataset curated by Google, this one focuses on sound and contains hundreds of thousands of 10-second samples broken down into categories such as instruments, vehicles, and vocals.
- Million Song Dataset - Samples and metadata from one million contemporary popular music tracks.
Text Data
- Wikidata - Database download of Wikipedia articles in many different formats.
- Common Crawl - An open data repository scraped from the World Wide Web, best known for training GPU large language models for ChatGPT and other chatbots.
Other and Miscellaneous Datasets
- Amazon Reviews - A database of approximately 35 million Amazon product reviews, including product information and ratings.
- Waymo Open Dataset – Alphabet’s self-driving subsidiary Waymo has disclosed a large amount of data collected through self-driving vehicles, including data from cameras and LiDAR sensors.
- Apolloscape Dataset——More autonomous driving data is provided by Baidu’s open source Apollo platform.
The above is the detailed content of Take stock of more than 20 powerful and free data sources that anyone can use to build AI. For more information, please follow other related articles on the PHP Chinese website!
Hot AI Tools
Undresser.AI Undress
AI-powered app for creating realistic nude photos
AI Clothes Remover
Online AI tool for removing clothes from photos.
Undress AI Tool
Undress images for free
Clothoff.io
AI clothes remover
Video Face Swap
Swap faces in any video effortlessly with our completely free AI face swap tool!
Hot Article
Hot Tools
Notepad++7.3.1
Easy-to-use and free code editor
SublimeText3 Chinese version
Chinese version, very easy to use
Zend Studio 13.0.1
Powerful PHP integrated development environment
Dreamweaver CS6
Visual web development tools
SublimeText3 Mac version
God-level code editing software (SublimeText3)
Hot Topics
1662
14
1419
52
1311
25
1261
29
1234
24
This article will take you to understand SHAP: model explanation for machine learning
Jun 01, 2024 am 10:58 AM
In the fields of machine learning and data science, model interpretability has always been a focus of researchers and practitioners. With the widespread application of complex models such as deep learning and ensemble methods, understanding the model's decision-making process has become particularly important. Explainable AI|XAI helps build trust and confidence in machine learning models by increasing the transparency of the model. Improving model transparency can be achieved through methods such as the widespread use of multiple complex models, as well as the decision-making processes used to explain the models. These methods include feature importance analysis, model prediction interval estimation, local interpretability algorithms, etc. Feature importance analysis can explain the decision-making process of a model by evaluating the degree of influence of the model on the input features. Model prediction interval estimate
Identify overfitting and underfitting through learning curves
Apr 29, 2024 pm 06:50 PM
This article will introduce how to effectively identify overfitting and underfitting in machine learning models through learning curves. Underfitting and overfitting 1. Overfitting If a model is overtrained on the data so that it learns noise from it, then the model is said to be overfitting. An overfitted model learns every example so perfectly that it will misclassify an unseen/new example. For an overfitted model, we will get a perfect/near-perfect training set score and a terrible validation set/test score. Slightly modified: "Cause of overfitting: Use a complex model to solve a simple problem and extract noise from the data. Because a small data set as a training set may not represent the correct representation of all data." 2. Underfitting Heru
The evolution of artificial intelligence in space exploration and human settlement engineering
Apr 29, 2024 pm 03:25 PM
In the 1950s, artificial intelligence (AI) was born. That's when researchers discovered that machines could perform human-like tasks, such as thinking. Later, in the 1960s, the U.S. Department of Defense funded artificial intelligence and established laboratories for further development. Researchers are finding applications for artificial intelligence in many areas, such as space exploration and survival in extreme environments. Space exploration is the study of the universe, which covers the entire universe beyond the earth. Space is classified as an extreme environment because its conditions are different from those on Earth. To survive in space, many factors must be considered and precautions must be taken. Scientists and researchers believe that exploring space and understanding the current state of everything can help understand how the universe works and prepare for potential environmental crises
Transparent! An in-depth analysis of the principles of major machine learning models!
Apr 12, 2024 pm 05:55 PM
In layman’s terms, a machine learning model is a mathematical function that maps input data to a predicted output. More specifically, a machine learning model is a mathematical function that adjusts model parameters by learning from training data to minimize the error between the predicted output and the true label. There are many models in machine learning, such as logistic regression models, decision tree models, support vector machine models, etc. Each model has its applicable data types and problem types. At the same time, there are many commonalities between different models, or there is a hidden path for model evolution. Taking the connectionist perceptron as an example, by increasing the number of hidden layers of the perceptron, we can transform it into a deep neural network. If a kernel function is added to the perceptron, it can be converted into an SVM. this one
Implementing Machine Learning Algorithms in C++: Common Challenges and Solutions
Jun 03, 2024 pm 01:25 PM
Common challenges faced by machine learning algorithms in C++ include memory management, multi-threading, performance optimization, and maintainability. Solutions include using smart pointers, modern threading libraries, SIMD instructions and third-party libraries, as well as following coding style guidelines and using automation tools. Practical cases show how to use the Eigen library to implement linear regression algorithms, effectively manage memory and use high-performance matrix operations.
Five schools of machine learning you don't know about
Jun 05, 2024 pm 08:51 PM
Machine learning is an important branch of artificial intelligence that gives computers the ability to learn from data and improve their capabilities without being explicitly programmed. Machine learning has a wide range of applications in various fields, from image recognition and natural language processing to recommendation systems and fraud detection, and it is changing the way we live. There are many different methods and theories in the field of machine learning, among which the five most influential methods are called the "Five Schools of Machine Learning". The five major schools are the symbolic school, the connectionist school, the evolutionary school, the Bayesian school and the analogy school. 1. Symbolism, also known as symbolism, emphasizes the use of symbols for logical reasoning and expression of knowledge. This school of thought believes that learning is a process of reverse deduction, through existing
Is Flash Attention stable? Meta and Harvard found that their model weight deviations fluctuated by orders of magnitude
May 30, 2024 pm 01:24 PM
MetaFAIR teamed up with Harvard to provide a new research framework for optimizing the data bias generated when large-scale machine learning is performed. It is known that the training of large language models often takes months and uses hundreds or even thousands of GPUs. Taking the LLaMA270B model as an example, its training requires a total of 1,720,320 GPU hours. Training large models presents unique systemic challenges due to the scale and complexity of these workloads. Recently, many institutions have reported instability in the training process when training SOTA generative AI models. They usually appear in the form of loss spikes. For example, Google's PaLM model experienced up to 20 loss spikes during the training process. Numerical bias is the root cause of this training inaccuracy,
Explainable AI: Explaining complex AI/ML models
Jun 03, 2024 pm 10:08 PM
Translator | Reviewed by Li Rui | Chonglou Artificial intelligence (AI) and machine learning (ML) models are becoming increasingly complex today, and the output produced by these models is a black box – unable to be explained to stakeholders. Explainable AI (XAI) aims to solve this problem by enabling stakeholders to understand how these models work, ensuring they understand how these models actually make decisions, and ensuring transparency in AI systems, Trust and accountability to address this issue. This article explores various explainable artificial intelligence (XAI) techniques to illustrate their underlying principles. Several reasons why explainable AI is crucial Trust and transparency: For AI systems to be widely accepted and trusted, users need to understand how decisions are made
