473,766 Members | 2,172 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

Efficient generation of random data subsets


I have a database table that contains a set of numbers (set X). I need to be
able to generate/export a random set of survey numbers (set C) from set A
with IDs
that aren't in set B. Where set A is a subset of numbers from set X that
match the filters for the survey, and set B contains the set of number IDs
for numbers that have already been exported for the survey.

The idea is to repeatedly generate a random sample of non-duplicate numbers
from set X filtering on the survey templates.

My database comprises of the following tables:

Table Numbers
ID INT PK
Number VARCHAR

Table Templates
ID INT PK
Template VARCHAR

Table Survey
ID INT PK
Name VARCHAR
Description VARCHAR
Started DATETIME

Table Filters
ID INT PK
SurveyID INT FK (Survey:ID)
TemplateID INT FK (Templates:ID)

Table NumberExportedS urveyJoin
ID INT PK
NumberID INT FK (Numbers:ID)
SurveyID INT FK (Survey:ID)
Exported DATETIME
This is the SQL query that I'm using to generate set C (the random set of
numbers) :

SELECT * FROM Numbers WHERE Number LIKE 'filter1%' OR Number LIKE 'filter2%'
AND Numbers.ID NOT IN (SELECT Numbers.ID FROM Numbers LEFT JOIN
NumberExportedS urveyJoin ON Numbers.ID = NumberExportedS urveyJoin.Numbe rID
WHERE NumberExportedS urveyJoin.Surve yID = S) ORDER BY RAND() LIMIT N;
The problem is that MySQL takes a long time to generate set C even for
moderate sizes of set A and set B. As the number of entries in table
NumberExportedS urveyJoin (set B) increases so does the time to generate the
next
random set of numbers. Is there a better/more efficient approach to
generating set C ?

Thanks,

Tony


Jul 20 '05 #1
0 1471

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

9
5610
by: Maurice LING | last post by:
Hi, Is there any UML tools that is able to take UML and generate Python codes? Cheers Maurice
4
13114
by: mescaline | last post by:
hi, i'm new to C++ could anyone refer me to a good site / good examples of random numbers? in particular including: 1) the commnds to obtain normally and exponenetially distributed r numbers
10
741
by: Ioannis Vranos | last post by:
I want to create some random numbers for encryption purposes, and i wonder if the following scheme makes it more hard to guess the underneath number generation pattern, than the plain use of rand(): #include <stdlib.h> #include <time.h>
13
3620
by: Roy Gourgi | last post by:
Hi, How do I invoke the random number generator that was suggested by a few people. Ideally, what I would like to do is to instantiate the random no. generator with a seed value that does not repeat the values and that can be called from any class, as I have to call the random number generator from a few different classes. Here is my code: using System;
4
4120
by: Dimos | last post by:
Hello All, I need some help with random number generation. What I need exactly is: To create a few thousand numbers, decimal and integers, between 5 and 90, and then to export them as a single column at a spreadsheet.
22
3454
by: gagan.singh.arora | last post by:
Hi there. I want to generate random numbers with a given probability, say 80% even and 20% odd. Is it possible to implement such an algorithm in C?
6
3439
by: py_genetic | last post by:
Hi, I'm looking to generate x alphabetic strings in a list size x. This is exactly the same output that the unix command "split" generates as default file name output when splitting large files. Example: produce x original, but not random strings from english alphabet, all lowercase. The length of each string and possible combinations is
4
3615
by: Patrick | last post by:
Hi, I want to write a programs that checks if a set of numbers in a list obey a condition, the problem is that i have say "n" numbers and i need to check all subsets of the n numbers for the condition. How do i go about asking c++ to find the subsets and then check?? Thanks: Patrick
3
3015
by: pierrej | last post by:
I was wondering if anyone could possibly assist me please. I'm going to come out and say I am not a C# dev and have limited knowledge on the subject but I am starting to learn it. I being developing a mail generation tool to pump dummy mail into an exchange environment for testing purposes. Everything works beautifully and the mail generation is grand but another requirement has come up where they want valid words (doesn't have to be...
0
9404
by: Hystou | last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can effortlessly switch the default language on Windows 10 without reinstalling. I'll walk you through it. First, let's disable language synchronization. With a Microsoft account, language settings sync across devices. To prevent any complications,...
0
10168
Oralloy
by: Oralloy | last post by:
Hello folks, I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>". The problem is that using the GNU compilers, it seems that the internal comparison operator "<=>" tries to promote arguments from unsigned to signed. This is as boiled down as I can make it. Here is my compilation command: g++-12 -std=c++20 -Wnarrowing bit_field.cpp Here is the code in...
0
10009
jinu1996
by: jinu1996 | last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven tapestry of website design and digital marketing. It's not merely about having a website; it's about crafting an immersive digital experience that captivates audiences and drives business growth. The Art of Business Website Design Your website is...
0
9838
tracyyun
by: tracyyun | last post by:
Dear forum friends, With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each protocol has its own unique characteristics and advantages, but as a user who is planning to build a smart home system, I am a bit confused by the choice of these technologies. I'm particularly interested in Zigbee because I've heard it does some...
0
8835
agi2029
by: agi2029 | last post by:
Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development project—planning, coding, testing, and deployment—without human intervention. Imagine an AI that can take a project description, break it down, write the code, debug it, and then launch it, all on its own.... Now, this would greatly impact the work of software developers. The idea...
1
7381
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 1 May 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome a new presenter, Adolph Dupré who will be discussing some powerful techniques for using class modules. He will explain when you may want to use classes instead of User Defined Types (UDT). For example, to manage the data in unbound forms. Adolph will...
0
6651
by: conductexam | last post by:
I have .net C# application in which I am extracting data from word file and save it in database particularly. To store word all data as it is I am converting the whole word file firstly in HTML and then checking html paragraph one by one. At the time of converting from word file to html my equations which are in the word document file was convert into image. Globals.ThisAddIn.Application.ActiveDocument.Select();...
1
3929
by: 6302768590 | last post by:
Hai team i want code for transfer the data from one system to another through IP address by using C# our system has to for every 5mins then we have to update the data what the data is updated we have to send another system
3
2806
bsmnconsultancy
by: bsmnconsultancy | last post by:
In today's digital era, a well-designed website is crucial for businesses looking to succeed. Whether you're a small business owner or a large corporation in Toronto, having a strong online presence can significantly impact your brand's success. BSMN Consultancy, a leader in Website Development in Toronto offers valuable insights into creating effective websites that not only look great but also perform exceptionally well. In this comprehensive...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.